Prevent curly braces from breaking in format strings

CHANGES.md

@@ -1,99 +1,3 @@
-Synapse 1.33.2 (2021-05-11)
-===========================
-
-Due to the security issue highlighted below, server administrators are encouraged to update Synapse. We are not aware of these vulnerabilities being exploited in the wild.
-
-Security advisory
------------------
-
-This release fixes a denial of service attack ([CVE-2021-29471](https://github.com/matrix-org/synapse/security/advisories/GHSA-x345-32rc-8h85)) against Synapse's push rules implementation. Server admins are encouraged to upgrade.
-
-Internal Changes
-----------------
-
-- Unpin attrs dependency. ([\#9946](https://github.com/matrix-org/synapse/issues/9946))
-
-
-Synapse 1.33.1 (2021-05-06)
-===========================
-
-Bugfixes
---------
-
-- Fix bug where `/sync` would break if using the latest version of `attrs` dependency, by pinning to a previous version. ([\#9937](https://github.com/matrix-org/synapse/issues/9937))
-
-
-Synapse 1.33.0 (2021-05-05)
-===========================
-
-Features
---------
-
-- Build Debian packages for Ubuntu 21.04 (Hirsute Hippo). ([\#9909](https://github.com/matrix-org/synapse/issues/9909))
-
-
-Synapse 1.33.0rc2 (2021-04-29)
-==============================
-
-Bugfixes
---------
-
-- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
-
-
-Synapse 1.33.0rc1 (2021-04-28)
-==============================
-
-Features
---------
-
-- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
-- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
-- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))
-
-
-Bugfixes
---------
-
-- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
-- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
-- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
-- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
-- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
-- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))
-
-
-Improved Documentation
-----------------------
-
-- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))
-
-
-Internal Changes
-----------------
-
-- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
-- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
-- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
-- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
-- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
-- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
-- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
-- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
-- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
-- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
-- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
-- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
-- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
-- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
-- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
-- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
-- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
-- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
-- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
-- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
-
-
 Synapse 1.32.2 (2021-04-22)
 ===========================
 

changelog.d/9162.misc

@@ -0,0 +1 @@
+Add a dockerfile for running Synapse in worker-mode under Complement.

changelog.d/9702.misc

@@ -0,0 +1 @@
+Speed up federation transmission by using fewer database calls. Contributed by @ShadowJonathan.

changelog.d/9726.bugfix

@@ -0,0 +1 @@
+Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path.

changelog.d/9786.misc

@@ -0,0 +1 @@
+Apply `pyupgrade` across the codebase.

changelog.d/9788.bugfix

@@ -0,0 +1 @@
+Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg.

changelog.d/9796.misc

@@ -0,0 +1 @@
+Move some replication processing out of `generic_worker`.

changelog.d/9800.feature

@@ -0,0 +1 @@
+Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership.

changelog.d/9801.doc

@@ -0,0 +1 @@
+Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms.

changelog.d/9802.bugfix

@@ -0,0 +1 @@
+Add some sanity checks to identity server passed to 3PID bind/unbind endpoints.

changelog.d/9814.feature

@@ -0,0 +1 @@
+Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership.

changelog.d/9815.misc

@@ -0,0 +1 @@
+Replace `HomeServer.get_config()` with inline references.

changelog.d/9816.misc

@@ -0,0 +1 @@
+Rename some handlers and config modules to not duplicate the top-level module.

changelog.d/9817.misc

@@ -0,0 +1 @@
+Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced.

changelog.d/9819.feature

@@ -0,0 +1 @@
+Add experimental support for handling presence on a worker.

changelog.d/9820.feature

@@ -0,0 +1 @@
+Add experimental support for handling presence on a worker.

changelog.d/9821.misc

@@ -0,0 +1 @@
+Reduce CPU usage of the user directory by reusing existing calculated room membership.

changelog.d/9825.misc

@@ -0,0 +1 @@
+Small speed up for joining large remote rooms.

changelog.d/9828.feature

@@ -0,0 +1 @@
+Add experimental support for handling presence on a worker.

changelog.d/9832.feature

@@ -0,0 +1 @@
+Don't return an error when a user attempts to renew their account multiple times with the same token. Instead, state when their account is set to expire. This change concerns the optional account validity feature.

changelog.d/9833.bugfix

@@ -0,0 +1 @@
+Limit the size of HTTP responses read over federation.

changelog.d/9838.misc

@@ -0,0 +1 @@
+Introduce flake8-bugbear to the test suite and fix some of its lint violations.

changelog.d/9845.misc

@@ -0,0 +1 @@
+Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores.

changelog.d/9850.feature

@@ -0,0 +1 @@
+Add experimental support for handling presence on a worker.

changelog.d/9855.misc

@@ -0,0 +1 @@
+Limit length of accepted email addresses.

changelog.d/9856.misc

@@ -0,0 +1 @@
+Remove redundant `synapse.types.Collection` type definition.

changelog.d/9858.misc

@@ -0,0 +1 @@
+Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts.

changelog.d/9867.bugfix

@@ -0,0 +1 @@
+Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists.

changelog.d/9871.misc

@@ -0,0 +1 @@
+Disable invite rate-limiting by default when running the unit tests.

changelog.d/9874.misc

@@ -0,0 +1 @@
+Pass a reactor into `SynapseSite` to make testing easier.

changelog.d/9875.misc

@@ -0,0 +1 @@
+Make `DomainSpecificString` an `attrs` class.

changelog.d/9876.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules.

changelog.d/9878.misc

@@ -0,0 +1 @@
+Remove redundant `_PushHTTPChannel` test class.

changelog.d/9887.misc

@@ -0,0 +1 @@
+Small performance improvement around handling new local presence updates.

contrib/experiments/test_messaging.py

@@ -224,14 +224,16 @@ class HomeServer(ReplicationHandler):
         destinations = yield self.get_servers_for_context(room_name)
 
         try:
-            yield self.replication_layer.send_pdu(
-                Pdu.create_new(
-                    context=room_name,
-                    pdu_type="sy.room.message",
-                    content={"sender": sender, "body": body},
-                    origin=self.server_name,
-                    destinations=destinations,
-                )
+            yield self.replication_layer.send_pdus(
+                [
+                    Pdu.create_new(
+                        context=room_name,
+                        pdu_type="sy.room.message",
+                        content={"sender": sender, "body": body},
+                        origin=self.server_name,
+                        destinations=destinations,
+                    )
+                ]
             )
         except Exception as e:
             logger.exception(e)
@@ -253,7 +255,7 @@ class HomeServer(ReplicationHandler):
                 origin=self.server_name,
                 destinations=destinations,
             )
-            yield self.replication_layer.send_pdu(pdu)
+            yield self.replication_layer.send_pdus([pdu])
         except Exception as e:
             logger.exception(e)
 
@@ -265,16 +267,18 @@ class HomeServer(ReplicationHandler):
         destinations = yield self.get_servers_for_context(room_name)
 
         try:
-            yield self.replication_layer.send_pdu(
-                Pdu.create_new(
-                    context=room_name,
-                    is_state=True,
-                    pdu_type="sy.room.member",
-                    state_key=invitee,
-                    content={"membership": "invite"},
-                    origin=self.server_name,
-                    destinations=destinations,
-                )
+            yield self.replication_layer.send_pdus(
+                [
+                    Pdu.create_new(
+                        context=room_name,
+                        is_state=True,
+                        pdu_type="sy.room.member",
+                        state_key=invitee,
+                        content={"membership": "invite"},
+                        origin=self.server_name,
+                        destinations=destinations,
+                    )
+                ]
             )
         except Exception as e:
             logger.exception(e)

debian/changelog

@@ -1,21 +1,3 @@
-matrix-synapse-py3 (1.33.2) stable; urgency=medium
-
-  * New synapse release 1.33.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 11 May 2021 11:17:59 +0100
-
-matrix-synapse-py3 (1.33.1) stable; urgency=medium
-
-  * New synapse release 1.33.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 06 May 2021 14:06:33 +0100
-
-matrix-synapse-py3 (1.33.0) stable; urgency=medium
-
-  * New synapse release 1.33.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 05 May 2021 14:15:27 +0100
-
 matrix-synapse-py3 (1.32.2) stable; urgency=medium
 
   * New synapse release 1.32.2.

docker/configure_workers_and_start.py

@@ -184,18 +184,18 @@ stderr_logfile_maxbytes=0
 """
 
 NGINX_LOCATION_CONFIG_BLOCK = """
-    location ~* {endpoint} {
+    location ~* {endpoint} {{
         proxy_pass {upstream};
         proxy_set_header X-Forwarded-For $remote_addr;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_set_header Host $host;
-    }
+    }}
 """
 
 NGINX_UPSTREAM_CONFIG_BLOCK = """
-upstream {upstream_worker_type} {
+upstream {upstream_worker_type} {{
 {body}
-}
+}}
 """
 
 

mypy.ini

@@ -41,6 +41,7 @@ files =
   synapse/push,
   synapse/replication,
   synapse/rest,
+  synapse/secrets.py,
   synapse/server.py,
   synapse/server_notices,
   synapse/spam_checker_api,

scripts-dev/build_debian_packages

@@ -21,10 +21,9 @@ DISTS = (
     "debian:buster",
     "debian:bullseye",
     "debian:sid",
-    "ubuntu:bionic",   # 18.04 LTS (our EOL forced by Py36 on 2021-12-23)
-    "ubuntu:focal",    # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
-    "ubuntu:groovy",   # 20.10 (EOL 2021-07-07)
-    "ubuntu:hirsute",  # 21.04 (EOL 2022-01-05)
+    "ubuntu:bionic",
+    "ubuntu:focal",
+    "ubuntu:groovy",
 )
 
 DESC = '''\

synapse/__init__.py

@@ -21,8 +21,8 @@ import os
 import sys
 
 # Check that we're not running on an unsupported Python version.
-if sys.version_info < (3, 6):
-    print("Synapse requires Python 3.6 or above.")
+if sys.version_info < (3, 5):
+    print("Synapse requires Python 3.5 or above.")
     sys.exit(1)
 
 # Twisted and canonicaljson will fail to import when this file is executed to
@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.33.2"
+__version__ = "1.32.2"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/config/tls.py

@@ -17,7 +17,7 @@ import os
 import warnings
 from datetime import datetime
 from hashlib import sha256
-from typing import List, Optional, Pattern
+from typing import List, Optional
 
 from unpaddedbase64 import encode_base64
 
@@ -124,7 +124,7 @@ class TlsConfig(Config):
             fed_whitelist_entries = []
 
         # Support globs (*) in whitelist values
-        self.federation_certificate_verification_whitelist = []  # type: List[Pattern]
+        self.federation_certificate_verification_whitelist = []  # type: List[str]
         for entry in fed_whitelist_entries:
             try:
                 entry_regex = glob_to_regex(entry.encode("ascii").decode("ascii"))

synapse/federation/federation_client.py

@@ -451,28 +451,6 @@ class FederationClient(FederationBase):
 
         return signed_auth
 
-    def _is_unknown_endpoint(
-        self, e: HttpResponseException, synapse_error: Optional[SynapseError] = None
-    ) -> bool:
-        """
-        Returns true if the response was due to an endpoint being unimplemented.
-
-        Args:
-            e: The error response received from the remote server.
-            synapse_error: The above error converted to a SynapseError. This is
-                automatically generated if not provided.
-
-        """
-        if synapse_error is None:
-            synapse_error = e.to_synapse_error()
-        # There is no good way to detect an "unknown" endpoint.
-        #
-        # Dendrite returns a 404 (with no body); synapse returns a 400
-        # with M_UNRECOGNISED.
-        return e.code == 404 or (
-            e.code == 400 and synapse_error.errcode == Codes.UNRECOGNIZED
-        )
-
     async def _try_destination_list(
         self,
         description: str,
@@ -490,9 +468,9 @@ class FederationClient(FederationBase):
             callback:  Function to run for each server. Passed a single
                 argument: the server_name to try.
 
-                If the callback raises a CodeMessageException with a 300/400 code or
-                an UnsupportedRoomVersionError, attempts to perform the operation
-                stop immediately and the exception is reraised.
+                If the callback raises a CodeMessageException with a 300/400 code,
+                attempts to perform the operation stop immediately and the exception is
+                reraised.
 
                 Otherwise, if the callback raises an Exception the error is logged and the
                 next server tried. Normally the stacktrace is logged but this is
@@ -514,7 +492,8 @@ class FederationClient(FederationBase):
                 continue
 
             try:
-                return await callback(destination)
+                res = await callback(destination)
+                return res
             except InvalidResponseError as e:
                 logger.warning("Failed to %s via %s: %s", description, destination, e)
             except UnsupportedRoomVersionError:
@@ -523,15 +502,17 @@ class FederationClient(FederationBase):
                 synapse_error = e.to_synapse_error()
                 failover = False
 
-                # Failover on an internal server error, or if the destination
-                # doesn't implemented the endpoint for some reason.
                 if 500 <= e.code < 600:
                     failover = True
 
-                elif failover_on_unknown_endpoint and self._is_unknown_endpoint(
-                    e, synapse_error
-                ):
-                    failover = True
+                elif failover_on_unknown_endpoint:
+                    # there is no good way to detect an "unknown" endpoint. Dendrite
+                    # returns a 404 (with no body); synapse returns a 400
+                    # with M_UNRECOGNISED.
+                    if e.code == 404 or (
+                        e.code == 400 and synapse_error.errcode == Codes.UNRECOGNIZED
+                    ):
+                        failover = True
 
                 if not failover:
                     raise synapse_error from e
@@ -589,8 +570,9 @@ class FederationClient(FederationBase):
             UnsupportedRoomVersionError: if remote responds with
                 a room version we don't understand.
 
-            SynapseError: if the chosen remote server returns a 300/400 code, or
-                no servers successfully handle the request.
+            SynapseError: if the chosen remote server returns a 300/400 code.
+
+            RuntimeError: if no servers were reachable.
         """
         valid_memberships = {Membership.JOIN, Membership.LEAVE}
         if membership not in valid_memberships:
@@ -660,8 +642,9 @@ class FederationClient(FederationBase):
             ``auth_chain``.
 
         Raises:
-            SynapseError: if the chosen remote server returns a 300/400 code, or
-                no servers successfully handle the request.
+            SynapseError: if the chosen remote server returns a 300/400 code.
+
+            RuntimeError: if no servers were reachable.
         """
 
         async def send_request(destination) -> Dict[str, Any]:
@@ -690,7 +673,7 @@ class FederationClient(FederationBase):
             if create_event is None:
                 # If the state doesn't have a create event then the room is
                 # invalid, and it would fail auth checks anyway.
-                raise InvalidResponseError("No create event in state")
+                raise SynapseError(400, "No create event in state")
 
             # the room version should be sane.
             create_room_version = create_event.content.get(
@@ -763,11 +746,16 @@ class FederationClient(FederationBase):
                 content=pdu.get_pdu_json(time_now),
             )
         except HttpResponseException as e:
-            # If an error is received that is due to an unrecognised endpoint,
-            # fallback to the v1 endpoint. Otherwise consider it a legitmate error
-            # and raise.
-            if not self._is_unknown_endpoint(e):
-                raise
+            if e.code in [400, 404]:
+                err = e.to_synapse_error()
+
+                # If we receive an error response that isn't a generic error, or an
+                # unrecognised endpoint error, we  assume that the remote understands
+                # the v2 invite API and this is a legitimate error.
+                if err.errcode not in [Codes.UNKNOWN, Codes.UNRECOGNIZED]:
+                    raise err
+            else:
+                raise e.to_synapse_error()
 
         logger.debug("Couldn't send_join with the v2 API, falling back to the v1 API")
 
@@ -814,11 +802,6 @@ class FederationClient(FederationBase):
 
         Returns:
             The event as a dict as returned by the remote server
-
-        Raises:
-            SynapseError: if the remote server returns an error or if the server
-                only supports the v1 endpoint and a room version other than "1"
-                or "2" is requested.
         """
         time_now = self._clock.time_msec()
 
@@ -834,19 +817,28 @@ class FederationClient(FederationBase):
                 },
             )
         except HttpResponseException as e:
-            # If an error is received that is due to an unrecognised endpoint,
-            # fallback to the v1 endpoint if the room uses old-style event IDs.
-            # Otherwise consider it a legitmate error and raise.
-            err = e.to_synapse_error()
-            if self._is_unknown_endpoint(e, err):
+            if e.code in [400, 404]:
+                err = e.to_synapse_error()
+
+                # If we receive an error response that isn't a generic error, we
+                # assume that the remote understands the v2 invite API and this
+                # is a legitimate error.
+                if err.errcode != Codes.UNKNOWN:
+                    raise err
+
+                # Otherwise, we assume that the remote server doesn't understand
+                # the v2 invite API. That's ok provided the room uses old-style event
+                # IDs.
                 if room_version.event_format != EventFormatVersions.V1:
                     raise SynapseError(
                         400,
                         "User's homeserver does not support this room version",
                         Codes.UNSUPPORTED_ROOM_VERSION,
                     )
+            elif e.code in (403, 429):
+                raise e.to_synapse_error()
             else:
-                raise err
+                raise
 
         # Didn't work, try v1 API.
         # Note the v1 API returns a tuple of `(200, content)`
@@ -873,8 +865,9 @@ class FederationClient(FederationBase):
             pdu: event to be sent
 
         Raises:
-            SynapseError: if the chosen remote server returns a 300/400 code, or
-                no servers successfully handle the request.
+            SynapseError if the chosen remote server returns a 300/400 code.
+
+            RuntimeError if no servers were reachable.
         """
 
         async def send_request(destination: str) -> None:
@@ -896,11 +889,16 @@ class FederationClient(FederationBase):
                 content=pdu.get_pdu_json(time_now),
             )
         except HttpResponseException as e:
-            # If an error is received that is due to an unrecognised endpoint,
-            # fallback to the v1 endpoint. Otherwise consider it a legitmate error
-            # and raise.
-            if not self._is_unknown_endpoint(e):
-                raise
+            if e.code in [400, 404]:
+                err = e.to_synapse_error()
+
+                # If we receive an error response that isn't a generic error, or an
+                # unrecognised endpoint error, we  assume that the remote understands
+                # the v2 invite API and this is a legitimate error.
+                if err.errcode not in [Codes.UNKNOWN, Codes.UNRECOGNIZED]:
+                    raise err
+            else:
+                raise e.to_synapse_error()
 
         logger.debug("Couldn't send_leave with the v2 API, falling back to the v1 API")
 

synapse/federation/sender/__init__.py

@@ -14,19 +14,26 @@
 
 import abc
 import logging
-from typing import TYPE_CHECKING, Dict, Hashable, Iterable, List, Optional, Set, Tuple
+from typing import (
+    TYPE_CHECKING,
+    Collection,
+    Dict,
+    Hashable,
+    Iterable,
+    List,
+    Optional,
+    Set,
+    Tuple,
+)
 
 from prometheus_client import Counter
 
-from twisted.internet import defer
-
 import synapse.metrics
 from synapse.api.presence import UserPresenceState
 from synapse.events import EventBase
 from synapse.federation.sender.per_destination_queue import PerDestinationQueue
 from synapse.federation.sender.transaction_manager import TransactionManager
 from synapse.federation.units import Edu
-from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.metrics import (
     LaterGauge,
     event_processing_loop_counter,
@@ -255,15 +262,27 @@ class FederationSender(AbstractFederationSender):
                 if not events and next_token >= self._last_poked_id:
                     break
 
-                async def handle_event(event: EventBase) -> None:
+                async def get_destinations_for_event(
+                    event: EventBase,
+                ) -> Collection[str]:
+                    """Computes the destinations to which this event must be sent.
+
+                    This returns an empty tuple when there are no destinations to send to,
+                    or if this event is not from this homeserver and it is not sending
+                    it on behalf of another server.
+
+                    Will also filter out destinations which this sender is not responsible for,
+                    if multiple federation senders exist.
+                    """
+
                     # Only send events for this server.
                     send_on_behalf_of = event.internal_metadata.get_send_on_behalf_of()
                     is_mine = self.is_mine_id(event.sender)
                     if not is_mine and send_on_behalf_of is None:
-                        return
+                        return ()
 
                     if not event.internal_metadata.should_proactively_send():
-                        return
+                        return ()
 
                     destinations = None  # type: Optional[Set[str]]
                     if not event.prev_event_ids():
@@ -298,7 +317,7 @@ class FederationSender(AbstractFederationSender):
                                 "Failed to calculate hosts in room for event: %s",
                                 event.event_id,
                             )
-                            return
+                            return ()
 
                     destinations = {
                         d
@@ -308,17 +327,15 @@ class FederationSender(AbstractFederationSender):
                         )
                     }
 
+                    destinations.discard(self.server_name)
+
                     if send_on_behalf_of is not None:
                         # If we are sending the event on behalf of another server
                         # then it already has the event and there is no reason to
                         # send the event to it.
                         destinations.discard(send_on_behalf_of)
 
-                    logger.debug("Sending %s to %r", event, destinations)
-
                     if destinations:
-                        await self._send_pdu(event, destinations)
-
                         now = self.clock.time_msec()
                         ts = await self.store.get_received_ts(event.event_id)
 
@@ -326,24 +343,29 @@ class FederationSender(AbstractFederationSender):
                             "federation_sender"
                         ).observe((now - ts) / 1000)
 
-                async def handle_room_events(events: Iterable[EventBase]) -> None:
-                    with Measure(self.clock, "handle_room_events"):
-                        for event in events:
-                            await handle_event(event)
+                        return destinations
+                    return ()
 
-                events_by_room = {}  # type: Dict[str, List[EventBase]]
-                for event in events:
-                    events_by_room.setdefault(event.room_id, []).append(event)
+                async def get_federatable_events_and_destinations(
+                    events: Iterable[EventBase],
+                ) -> List[Tuple[EventBase, Collection[str]]]:
+                    with Measure(self.clock, "get_destinations_for_events"):
+                        # Fetch federation destinations per event,
+                        # skip if get_destinations_for_event returns an empty collection,
+                        # return list of event->destinations pairs.
+                        return [
+                            (event, dests)
+                            for (event, dests) in [
+                                (event, await get_destinations_for_event(event))
+                                for event in events
+                            ]
+                            if dests
+                        ]
 
-                await make_deferred_yieldable(
-                    defer.gatherResults(
-                        [
-                            run_in_background(handle_room_events, evs)
-                            for evs in events_by_room.values()
-                        ],
-                        consumeErrors=True,
-                    )
-                )
+                events_and_dests = await get_federatable_events_and_destinations(events)
+
+                # Send corresponding events to each destination queue
+                await self._distribute_events(events_and_dests)
 
                 await self.store.update_federation_out_pos("events", next_token)
 
@@ -361,7 +383,7 @@ class FederationSender(AbstractFederationSender):
                     events_processed_counter.inc(len(events))
 
                     event_processing_loop_room_count.labels("federation_sender").inc(
-                        len(events_by_room)
+                        len({event.room_id for event in events})
                     )
 
                 event_processing_loop_counter.labels("federation_sender").inc()
@@ -373,34 +395,53 @@ class FederationSender(AbstractFederationSender):
         finally:
             self._is_processing = False
 
-    async def _send_pdu(self, pdu: EventBase, destinations: Iterable[str]) -> None:
-        # We loop through all destinations to see whether we already have
-        # a transaction in progress. If we do, stick it in the pending_pdus
-        # table and we'll get back to it later.
+    async def _distribute_events(
+        self,
+        events_and_dests: Iterable[Tuple[EventBase, Collection[str]]],
+    ) -> None:
+        """Distribute events to the respective per_destination queues.
 
-        destinations = set(destinations)
-        destinations.discard(self.server_name)
-        logger.debug("Sending to: %s", str(destinations))
+        Also persists last-seen per-room stream_ordering to 'destination_rooms'.
 
-        if not destinations:
-            return
+        Args:
+            events_and_dests: A list of tuples, which are (event: EventBase, destinations: Collection[str]).
+                              Every event is paired with its intended destinations (in federation).
+        """
+        # Tuples of room_id + destination to their max-seen stream_ordering
+        room_with_dest_stream_ordering = {}  # type: Dict[Tuple[str, str], int]
 
-        sent_pdus_destination_dist_total.inc(len(destinations))
-        sent_pdus_destination_dist_count.inc()
+        # List of events to send to each destination
+        events_by_dest = {}  # type: Dict[str, List[EventBase]]
 
-        assert pdu.internal_metadata.stream_ordering
+        # For each event-destinations pair...
+        for event, destinations in events_and_dests:
 
-        # track the fact that we have a PDU for these destinations,
-        # to allow us to perform catch-up later on if the remote is unreachable
-        # for a while.
-        await self.store.store_destination_rooms_entries(
-            destinations,
-            pdu.room_id,
-            pdu.internal_metadata.stream_ordering,
+            # (we got this from the database, it's filled)
+            assert event.internal_metadata.stream_ordering
+
+            sent_pdus_destination_dist_total.inc(len(destinations))
+            sent_pdus_destination_dist_count.inc()
+
+            # ...iterate over those destinations..
+            for destination in destinations:
+                # ...update their stream-ordering...
+                room_with_dest_stream_ordering[(event.room_id, destination)] = max(
+                    event.internal_metadata.stream_ordering,
+                    room_with_dest_stream_ordering.get((event.room_id, destination), 0),
+                )
+
+                # ...and add the event to each destination queue.
+                events_by_dest.setdefault(destination, []).append(event)
+
+        # Bulk-store destination_rooms stream_ids
+        await self.store.bulk_store_destination_rooms_entries(
+            room_with_dest_stream_ordering
         )
 
-        for destination in destinations:
-            self._get_per_destination_queue(destination).send_pdu(pdu)
+        for destination, pdus in events_by_dest.items():
+            logger.debug("Sending %d pdus to %s", len(pdus), destination)
+
+            self._get_per_destination_queue(destination).send_pdus(pdus)
 
     async def send_read_receipt(self, receipt: ReadReceipt) -> None:
         """Send a RR to any other servers in the room

synapse/federation/sender/per_destination_queue.py

@@ -154,19 +154,22 @@ class PerDestinationQueue:
             + len(self._pending_edus_keyed)
         )
 
-    def send_pdu(self, pdu: EventBase) -> None:
-        """Add a PDU to the queue, and start the transmission loop if necessary
+    def send_pdus(self, pdus: Iterable[EventBase]) -> None:
+        """Add PDUs to the queue, and start the transmission loop if necessary
 
         Args:
-            pdu: pdu to send
+            pdus: pdus to send
         """
         if not self._catching_up or self._last_successful_stream_ordering is None:
             # only enqueue the PDU if we are not catching up (False) or do not
             # yet know if we have anything to catch up (None)
-            self._pending_pdus.append(pdu)
+            self._pending_pdus.extend(pdus)
         else:
-            assert pdu.internal_metadata.stream_ordering
-            self._catchup_last_skipped = pdu.internal_metadata.stream_ordering
+            self._catchup_last_skipped = max(
+                pdu.internal_metadata.stream_ordering
+                for pdu in pdus
+                if pdu.internal_metadata.stream_ordering is not None
+            )
 
         self.attempt_new_transaction()
 

synapse/handlers/presence.py

@@ -2026,40 +2026,18 @@ class PresenceFederationQueue:
             )
             return result["updates"], result["upto_token"], result["limited"]
 
-        # If the from_token is the current token then there's nothing to return
-        # and we can trivially no-op.
-        if from_token == self._next_id - 1:
-            return [], upto_token, False
-
         # We can find the correct position in the queue by noting that there is
         # exactly one entry per stream ID, and that the last entry has an ID of
         # `self._next_id - 1`, so we can count backwards from the end.
         #
-        # Since we are returning all states in the range `from_token < stream_id
-        # <= upto_token` we look for the index with a `stream_id` of `from_token
-        # + 1`.
-        #
         # Since the start of the queue is periodically truncated we need to
         # handle the case where `from_token` stream ID has already been dropped.
-        start_idx = max(from_token + 1 - self._next_id, -len(self._queue))
+        start_idx = max(from_token - self._next_id, -len(self._queue))
 
         to_send = []  # type: List[Tuple[int, Tuple[str, str]]]
         limited = False
         new_id = upto_token
         for _, stream_id, destinations, user_ids in self._queue[start_idx:]:
-            if stream_id <= from_token:
-                # Paranoia check that we are actually only sending states that
-                # are have stream_id strictly greater than from_token. We should
-                # never hit this.
-                logger.warning(
-                    "Tried returning presence federation stream ID: %d less than from_token: %d (next_id: %d, len: %d)",
-                    stream_id,
-                    from_token,
-                    self._next_id,
-                    len(self._queue),
-                )
-                continue
-
             if stream_id > upto_token:
                 break
 

synapse/push/push_rule_evaluator.py

@@ -19,7 +19,6 @@ from typing import Any, Dict, List, Optional, Pattern, Tuple, Union
 
 from synapse.events import EventBase
 from synapse.types import UserID
-from synapse.util import glob_to_regex, re_word_boundary
 from synapse.util.caches.lrucache import LruCache
 
 logger = logging.getLogger(__name__)
@@ -184,7 +183,7 @@ class PushRuleEvaluatorForEvent:
         r = regex_cache.get((display_name, False, True), None)
         if not r:
             r1 = re.escape(display_name)
-            r1 = re_word_boundary(r1)
+            r1 = _re_word_boundary(r1)
             r = re.compile(r1, flags=re.IGNORECASE)
             regex_cache[(display_name, False, True)] = r
 
@@ -213,7 +212,7 @@ def _glob_matches(glob: str, value: str, word_boundary: bool = False) -> bool:
     try:
         r = regex_cache.get((glob, True, word_boundary), None)
         if not r:
-            r = glob_to_regex(glob, word_boundary)
+            r = _glob_to_re(glob, word_boundary)
             regex_cache[(glob, True, word_boundary)] = r
         return bool(r.search(value))
     except re.error:
@@ -221,6 +220,56 @@ def _glob_matches(glob: str, value: str, word_boundary: bool = False) -> bool:
         return False
 
 
+def _glob_to_re(glob: str, word_boundary: bool) -> Pattern:
+    """Generates regex for a given glob.
+
+    Args:
+        glob
+        word_boundary: Whether to match against word boundaries or entire string.
+    """
+    if IS_GLOB.search(glob):
+        r = re.escape(glob)
+
+        r = r.replace(r"\*", ".*?")
+        r = r.replace(r"\?", ".")
+
+        # handle [abc], [a-z] and [!a-z] style ranges.
+        r = GLOB_REGEX.sub(
+            lambda x: (
+                "[%s%s]" % (x.group(1) and "^" or "", x.group(2).replace(r"\\\-", "-"))
+            ),
+            r,
+        )
+        if word_boundary:
+            r = _re_word_boundary(r)
+
+            return re.compile(r, flags=re.IGNORECASE)
+        else:
+            r = "^" + r + "$"
+
+            return re.compile(r, flags=re.IGNORECASE)
+    elif word_boundary:
+        r = re.escape(glob)
+        r = _re_word_boundary(r)
+
+        return re.compile(r, flags=re.IGNORECASE)
+    else:
+        r = "^" + re.escape(glob) + "$"
+        return re.compile(r, flags=re.IGNORECASE)
+
+
+def _re_word_boundary(r: str) -> str:
+    """
+    Adds word boundary characters to the start and end of an
+    expression to require that the match occur as a whole word,
+    but do so respecting the fact that strings starting or ending
+    with non-word characters will change word boundaries.
+    """
+    # we can't use \b as it chokes on unicode. however \W seems to be okay
+    # as shorthand for [^0-9A-Za-z_].
+    return r"(^|\W)%s(\W|$)" % (r,)
+
+
 def _flatten_dict(
     d: Union[EventBase, dict],
     prefix: Optional[List[str]] = None,

synapse/python_dependencies.py

@@ -78,15 +78,14 @@ REQUIREMENTS = [
     # we use attr.validators.deep_iterable, which arrived in 19.1.0 (Note:
     # Fedora 31 only has 19.1, so if we want to upgrade we should wait until 33
     # is out in November.)
-    # Note: 21.1.0 broke `/sync`, see #9936
-    "attrs>=19.1.0,!=21.1.0",
+    "attrs>=19.1.0",
     "netaddr>=0.7.18",
     "Jinja2>=2.9",
     "bleach>=1.4.3",
     "typing-extensions>=3.7.4",
     # We enforce that we have a `cryptography` version that bundles an `openssl`
     # with the latest security patches.
-    "cryptography>=3.4.7",
+    "cryptography>=3.4.7;python_version>='3.6'",
 ]
 
 CONDITIONAL_REQUIREMENTS = {
@@ -101,9 +100,14 @@ CONDITIONAL_REQUIREMENTS = {
     # that use the protocol, such as Let's Encrypt.
     "acme": [
         "txacme>=0.9.2",
+        # txacme depends on eliot. Eliot 1.8.0 is incompatible with
+        # python 3.5.2, as per https://github.com/itamarst/eliot/issues/418
+        "eliot<1.8.0;python_version<'3.5.3'",
     ],
     "saml2": [
-        "pysaml2>=4.5.0",
+        # pysaml2 6.4.0 is incompatible with Python 3.5 (see https://github.com/IdentityPython/pysaml2/issues/749)
+        "pysaml2>=4.5.0,<6.4.0;python_version<'3.6'",
+        "pysaml2>=4.5.0;python_version>='3.6'",
     ],
     "oidc": ["authlib>=0.14.0"],
     # systemd-python is necessary for logging to the systemd journal via

synapse/rest/admin/users.py

@@ -14,7 +14,6 @@
 import hashlib
 import hmac
 import logging
-import secrets
 from http import HTTPStatus
 from typing import TYPE_CHECKING, Dict, List, Optional, Tuple
 
@@ -376,7 +375,7 @@ class UserRegisterServlet(RestServlet):
         """
         self._clear_old_nonces()
 
-        nonce = secrets.token_hex(64)
+        nonce = self.hs.get_secrets().token_hex(64)
         self.nonces[nonce] = int(self.reactor.seconds())
         return 200, {"nonce": nonce}
 

synapse/rest/consent/consent_resource.py

@@ -32,6 +32,14 @@ TEMPLATE_LANGUAGE = "en"
 
 logger = logging.getLogger(__name__)
 
+# use hmac.compare_digest if we have it (python 2.7.7), else just use equality
+if hasattr(hmac, "compare_digest"):
+    compare_digest = hmac.compare_digest
+else:
+
+    def compare_digest(a, b):
+        return a == b
+
 
 class ConsentResource(DirectServeHtmlResource):
     """A twisted Resource to display a privacy policy and gather consent to it
@@ -201,5 +209,5 @@ class ConsentResource(DirectServeHtmlResource):
             .encode("ascii")
         )
 
-        if not hmac.compare_digest(want_mac, userhmac):
+        if not compare_digest(want_mac, userhmac):
             raise SynapseError(HTTPStatus.FORBIDDEN, "HMAC incorrect")

synapse/rest/media/v1/filepath.py

@@ -21,7 +21,7 @@ from typing import Callable, List
 NEW_FORMAT_ID_RE = re.compile(r"^\d\d\d\d-\d\d-\d\d")
 
 
-def _wrap_in_base_path(func: Callable[..., str]) -> Callable[..., str]:
+def _wrap_in_base_path(func: "Callable[..., str]") -> "Callable[..., str]":
     """Takes a function that returns a relative path and turns it into an
     absolute path based on the location of the primary media store
     """

synapse/secrets.py

@@ -0,0 +1,44 @@
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Injectable secrets module for Synapse.
+
+See https://docs.python.org/3/library/secrets.html#module-secrets for the API
+used in Python 3.6, and the API emulated in Python 2.7.
+"""
+import sys
+
+# secrets is available since python 3.6
+if sys.version_info[0:2] >= (3, 6):
+    import secrets
+
+    class Secrets:
+        def token_bytes(self, nbytes: int = 32) -> bytes:
+            return secrets.token_bytes(nbytes)
+
+        def token_hex(self, nbytes: int = 32) -> str:
+            return secrets.token_hex(nbytes)
+
+
+else:
+    import binascii
+    import os
+
+    class Secrets:
+        def token_bytes(self, nbytes: int = 32) -> bytes:
+            return os.urandom(nbytes)
+
+        def token_hex(self, nbytes: int = 32) -> str:
+            return binascii.hexlify(self.token_bytes(nbytes)).decode("ascii")

synapse/server.py

@@ -126,6 +126,7 @@ from synapse.rest.media.v1.media_repository import (
     MediaRepository,
     MediaRepositoryResource,
 )
+from synapse.secrets import Secrets
 from synapse.server_notices.server_notices_manager import ServerNoticesManager
 from synapse.server_notices.server_notices_sender import ServerNoticesSender
 from synapse.server_notices.worker_server_notices_sender import (
@@ -640,6 +641,10 @@ class HomeServer(metaclass=abc.ABCMeta):
     def get_groups_attestation_renewer(self) -> GroupAttestionRenewer:
         return GroupAttestionRenewer(self)
 
+    @cache_in_self
+    def get_secrets(self) -> Secrets:
+        return Secrets()
+
     @cache_in_self
     def get_stats_handler(self) -> StatsHandler:
         return StatsHandler(self)

synapse/storage/_base.py

@@ -114,7 +114,7 @@ def db_to_json(db_content: Union[memoryview, bytes, bytearray, str]) -> Any:
         db_content = db_content.tobytes()
 
     # Decode it to a Unicode string before feeding it to the JSON decoder, since
-    # it only supports handling strings
+    # Python 3.5 does not support deserializing bytes.
     if isinstance(db_content, (bytes, bytearray)):
         db_content = db_content.decode("utf8")
 

synapse/storage/database.py

@@ -171,7 +171,10 @@ class LoggingDatabaseConnection:
 
 
 # The type of entry which goes on our after_callbacks and exception_callbacks lists.
-_CallbackListEntry = Tuple[Callable[..., None], Iterable[Any], Dict[str, Any]]
+#
+# Python 3.5.2 doesn't support Callable with an ellipsis, so we wrap it in quotes so
+# that mypy sees the type but the runtime python doesn't.
+_CallbackListEntry = Tuple["Callable[..., None]", Iterable[Any], Dict[str, Any]]
 
 
 R = TypeVar("R")
@@ -218,7 +221,7 @@ class LoggingTransaction:
         self.after_callbacks = after_callbacks
         self.exception_callbacks = exception_callbacks
 
-    def call_after(self, callback: Callable[..., None], *args: Any, **kwargs: Any):
+    def call_after(self, callback: "Callable[..., None]", *args: Any, **kwargs: Any):
         """Call the given callback on the main twisted thread after the
         transaction has finished. Used to invalidate the caches on the
         correct thread.
@@ -230,7 +233,7 @@ class LoggingTransaction:
         self.after_callbacks.append((callback, args, kwargs))
 
     def call_on_exception(
-        self, callback: Callable[..., None], *args: Any, **kwargs: Any
+        self, callback: "Callable[..., None]", *args: Any, **kwargs: Any
     ):
         # if self.exception_callbacks is None, that means that whatever constructed the
         # LoggingTransaction isn't expecting there to be any callbacks; assert that
@@ -482,7 +485,7 @@ class DatabasePool:
         desc: str,
         after_callbacks: List[_CallbackListEntry],
         exception_callbacks: List[_CallbackListEntry],
-        func: Callable[..., R],
+        func: "Callable[..., R]",
         *args: Any,
         **kwargs: Any,
     ) -> R:
@@ -615,7 +618,7 @@ class DatabasePool:
     async def runInteraction(
         self,
         desc: str,
-        func: Callable[..., R],
+        func: "Callable[..., R]",
         *args: Any,
         db_autocommit: bool = False,
         **kwargs: Any,
@@ -675,7 +678,7 @@ class DatabasePool:
 
     async def runWithConnection(
         self,
-        func: Callable[..., R],
+        func: "Callable[..., R]",
         *args: Any,
         db_autocommit: bool = False,
         **kwargs: Any,

synapse/storage/databases/main/transactions.py

@@ -14,7 +14,7 @@
 
 import logging
 from collections import namedtuple
-from typing import Iterable, List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple
 
 from canonicaljson import encode_canonical_json
 
@@ -295,37 +295,33 @@ class TransactionStore(TransactionWorkerStore):
                 },
             )
 
-    async def store_destination_rooms_entries(
-        self,
-        destinations: Iterable[str],
-        room_id: str,
-        stream_ordering: int,
-    ) -> None:
+    async def bulk_store_destination_rooms_entries(
+        self, room_and_destination_to_ordering: Dict[Tuple[str, str], int]
+    ):
         """
-        Updates or creates `destination_rooms` entries in batch for a single event.
+        Updates or creates `destination_rooms` entries for a number of events.
 
         Args:
-            destinations: list of destinations
-            room_id: the room_id of the event
-            stream_ordering: the stream_ordering of the event
+            room_and_destination_to_ordering: A mapping of (room, destination) -> stream_id
         """
 
         await self.db_pool.simple_upsert_many(
             table="destinations",
             key_names=("destination",),
-            key_values=[(d,) for d in destinations],
+            key_values={(d,) for _, d in room_and_destination_to_ordering.keys()},
             value_names=[],
             value_values=[],
             desc="store_destination_rooms_entries_dests",
         )
 
-        rows = [(destination, room_id) for destination in destinations]
         await self.db_pool.simple_upsert_many(
             table="destination_rooms",
-            key_names=("destination", "room_id"),
-            key_values=rows,
+            key_names=("room_id", "destination"),
+            key_values=list(room_and_destination_to_ordering.keys()),
             value_names=["stream_ordering"],
-            value_values=[(stream_ordering,)] * len(rows),
+            value_values=[
+                (stream_id,) for stream_id in room_and_destination_to_ordering.values()
+            ],
             desc="store_destination_rooms_entries_rooms",
         )
 

synapse/util/__init__.py

@@ -15,7 +15,6 @@
 import json
 import logging
 import re
-from typing import Pattern
 
 import attr
 from frozendict import frozendict
@@ -27,9 +26,6 @@ from synapse.logging import context
 logger = logging.getLogger(__name__)
 
 
-_WILDCARD_RUN = re.compile(r"([\?\*]+)")
-
-
 def _reject_invalid_json(val):
     """Do not allow Infinity, -Infinity, or NaN values in JSON."""
     raise ValueError("Invalid JSON value: '%s'" % val)
@@ -162,54 +158,25 @@ def log_failure(failure, msg, consumeErrors=True):
         return failure
 
 
-def glob_to_regex(glob: str, word_boundary: bool = False) -> Pattern:
+def glob_to_regex(glob):
     """Converts a glob to a compiled regex object.
 
+    The regex is anchored at the beginning and end of the string.
+
     Args:
-        glob: pattern to match
-        word_boundary: If True, the pattern will be allowed to match at word boundaries
-           anywhere in the string. Otherwise, the pattern is anchored at the start and
-           end of the string.
+        glob (str)
 
     Returns:
-        compiled regex pattern
+        re.RegexObject
     """
-
-    # Patterns with wildcards must be simplified to avoid performance cliffs
-    # - The glob `?**?**?` is equivalent to the glob `???*`
-    # - The glob `???*` is equivalent to the regex `.{3,}`
-    chunks = []
-    for chunk in _WILDCARD_RUN.split(glob):
-        # No wildcards? re.escape()
-        if not _WILDCARD_RUN.match(chunk):
-            chunks.append(re.escape(chunk))
-            continue
-
-        # Wildcards? Simplify.
-        qmarks = chunk.count("?")
-        if "*" in chunk:
-            chunks.append(".{%d,}" % qmarks)
+    res = ""
+    for c in glob:
+        if c == "*":
+            res = res + ".*"
+        elif c == "?":
+            res = res + "."
         else:
-            chunks.append(".{%d}" % qmarks)
+            res = res + re.escape(c)
 
-    res = "".join(chunks)
-
-    if word_boundary:
-        res = re_word_boundary(res)
-    else:
-        # \A anchors at start of string, \Z at end of string
-        res = r"\A" + res + r"\Z"
-
-    return re.compile(res, re.IGNORECASE)
-
-
-def re_word_boundary(r: str) -> str:
-    """
-    Adds word boundary characters to the start and end of an
-    expression to require that the match occur as a whole word,
-    but do so respecting the fact that strings starting or ending
-    with non-word characters will change word boundaries.
-    """
-    # we can't use \b as it chokes on unicode. however \W seems to be okay
-    # as shorthand for [^0-9A-Za-z_].
-    return r"(^|\W)%s(\W|$)" % (r,)
+    # \A anchors at start of string, \Z at end of string
+    return re.compile(r"\A" + res + r"\Z", re.IGNORECASE)

synapse/util/caches/response_cache.py

@@ -110,7 +110,7 @@ class ResponseCache(Generic[T]):
         return result.observe()
 
     def wrap(
-        self, key: T, callback: Callable[..., Any], *args: Any, **kwargs: Any
+        self, key: T, callback: "Callable[..., Any]", *args: Any, **kwargs: Any
     ) -> defer.Deferred:
         """Wrap together a *get* and *set* call, taking care of logcontexts
 

tests/federation/test_federation_server.py

@@ -74,25 +74,6 @@ class ServerACLsTestCase(unittest.TestCase):
         self.assertFalse(server_matches_acl_event("[1:2::]", e))
         self.assertTrue(server_matches_acl_event("1:2:3:4", e))
 
-    def test_wildcard_matching(self):
-        e = _create_acl_event({"allow": ["good*.com"]})
-        self.assertTrue(
-            server_matches_acl_event("good.com", e),
-            "* matches 0 characters",
-        )
-        self.assertTrue(
-            server_matches_acl_event("GOOD.COM", e),
-            "pattern is case-insensitive",
-        )
-        self.assertTrue(
-            server_matches_acl_event("good.aa.com", e),
-            "* matches several characters, including '.'",
-        )
-        self.assertFalse(
-            server_matches_acl_event("ishgood.com", e),
-            "pattern does not allow prefixes",
-        )
-
 
 class StateQueryTests(unittest.FederatingHomeserverTestCase):
 

tests/handlers/test_presence.py

@@ -509,14 +509,6 @@ class PresenceFederationQueueTestCase(unittest.HomeserverTestCase):
 
         self.assertCountEqual(rows, expected_rows)
 
-        now_token = self.queue.get_current_token(self.instance_name)
-        rows, upto_token, limited = self.get_success(
-            self.queue.get_replication_rows("master", upto_token, now_token, 10)
-        )
-        self.assertEqual(upto_token, now_token)
-        self.assertFalse(limited)
-        self.assertCountEqual(rows, [])
-
     def test_send_and_get_split(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")
@@ -546,20 +538,6 @@ class PresenceFederationQueueTestCase(unittest.HomeserverTestCase):
 
         self.assertCountEqual(rows, expected_rows)
 
-        now_token = self.queue.get_current_token(self.instance_name)
-        rows, upto_token, limited = self.get_success(
-            self.queue.get_replication_rows("master", upto_token, now_token, 10)
-        )
-
-        self.assertEqual(upto_token, now_token)
-        self.assertFalse(limited)
-
-        expected_rows = [
-            (2, ("dest3", "@user3:test")),
-        ]
-
-        self.assertCountEqual(rows, expected_rows)
-
     def test_clear_queue_all(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")

tests/push/test_push_rule_evaluator.py

@@ -12,8 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from typing import Any, Dict
-
 from synapse.api.room_versions import RoomVersions
 from synapse.events import FrozenEvent
 from synapse.push import push_rule_evaluator
@@ -68,170 +66,6 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
         # A display name with spaces should work fine.
         self.assertTrue(evaluator.matches(condition, "@user:test", "foo bar"))
 
-    def _assert_matches(
-        self, condition: Dict[str, Any], content: Dict[str, Any], msg=None
-    ) -> None:
-        evaluator = self._get_evaluator(content)
-        self.assertTrue(evaluator.matches(condition, "@user:test", "display_name"), msg)
-
-    def _assert_not_matches(
-        self, condition: Dict[str, Any], content: Dict[str, Any], msg=None
-    ) -> None:
-        evaluator = self._get_evaluator(content)
-        self.assertFalse(
-            evaluator.matches(condition, "@user:test", "display_name"), msg
-        )
-
-    def test_event_match_body(self):
-        """Check that event_match conditions on content.body work as expected"""
-
-        # if the key is `content.body`, the pattern matches substrings.
-
-        # non-wildcards should match
-        condition = {
-            "kind": "event_match",
-            "key": "content.body",
-            "pattern": "foobaz",
-        }
-        self._assert_matches(
-            condition,
-            {"body": "aaa FoobaZ zzz"},
-            "patterns should match and be case-insensitive",
-        )
-        self._assert_not_matches(
-            condition,
-            {"body": "aa xFoobaZ yy"},
-            "pattern should only match at word boundaries",
-        )
-        self._assert_not_matches(
-            condition,
-            {"body": "aa foobazx yy"},
-            "pattern should only match at word boundaries",
-        )
-
-        # wildcards should match
-        condition = {
-            "kind": "event_match",
-            "key": "content.body",
-            "pattern": "f?o*baz",
-        }
-
-        self._assert_matches(
-            condition,
-            {"body": "aaa FoobarbaZ zzz"},
-            "* should match string and pattern should be case-insensitive",
-        )
-        self._assert_matches(
-            condition, {"body": "aa foobaz yy"}, "* should match 0 characters"
-        )
-        self._assert_not_matches(
-            condition, {"body": "aa fobbaz yy"}, "? should not match 0 characters"
-        )
-        self._assert_not_matches(
-            condition, {"body": "aa fiiobaz yy"}, "? should not match 2 characters"
-        )
-        self._assert_not_matches(
-            condition,
-            {"body": "aa xfooxbaz yy"},
-            "pattern should only match at word boundaries",
-        )
-        self._assert_not_matches(
-            condition,
-            {"body": "aa fooxbazx yy"},
-            "pattern should only match at word boundaries",
-        )
-
-        # test backslashes
-        condition = {
-            "kind": "event_match",
-            "key": "content.body",
-            "pattern": r"f\oobaz",
-        }
-        self._assert_matches(
-            condition,
-            {"body": r"F\oobaz"},
-            "backslash should match itself",
-        )
-        condition = {
-            "kind": "event_match",
-            "key": "content.body",
-            "pattern": r"f\?obaz",
-        }
-        self._assert_matches(
-            condition,
-            {"body": r"F\oobaz"},
-            r"? after \ should match any character",
-        )
-
-    def test_event_match_non_body(self):
-        """Check that event_match conditions on other keys work as expected"""
-
-        # if the key is anything other than 'content.body', the pattern must match the
-        # whole value.
-
-        # non-wildcards should match
-        condition = {
-            "kind": "event_match",
-            "key": "content.value",
-            "pattern": "foobaz",
-        }
-        self._assert_matches(
-            condition,
-            {"value": "FoobaZ"},
-            "patterns should match and be case-insensitive",
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "xFoobaZ"},
-            "pattern should only match at the start/end of the value",
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "FoobaZz"},
-            "pattern should only match at the start/end of the value",
-        )
-
-        # wildcards should match
-        condition = {
-            "kind": "event_match",
-            "key": "content.value",
-            "pattern": "f?o*baz",
-        }
-        self._assert_matches(
-            condition,
-            {"value": "FoobarbaZ"},
-            "* should match string and pattern should be case-insensitive",
-        )
-        self._assert_matches(
-            condition, {"value": "foobaz"}, "* should match 0 characters"
-        )
-        self._assert_not_matches(
-            condition, {"value": "fobbaz"}, "? should not match 0 characters"
-        )
-        self._assert_not_matches(
-            condition, {"value": "fiiobaz"}, "? should not match 2 characters"
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "xfooxbaz"},
-            "pattern should only match at the start/end of the value",
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "fooxbazx"},
-            "pattern should only match at the start/end of the value",
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "x\nfooxbaz"},
-            "pattern should not match after a newline",
-        )
-        self._assert_not_matches(
-            condition,
-            {"value": "fooxbaz\nx"},
-            "pattern should not match before a newline",
-        )
-
     def test_no_body(self):
         """Not having a body shouldn't break the evaluator."""
         evaluator = self._get_evaluator({})

tests/rest/admin/test_user.py

@@ -18,7 +18,7 @@ import json
 import urllib.parse
 from binascii import unhexlify
 from typing import List, Optional
-from unittest.mock import Mock, patch
+from unittest.mock import Mock
 
 import synapse.rest.admin
 from synapse.api.constants import UserTypes
@@ -54,6 +54,8 @@ class UserRegisterTestCase(unittest.HomeserverTestCase):
         self.datastore = Mock(return_value=Mock())
         self.datastore.get_current_state_deltas = Mock(return_value=(0, []))
 
+        self.secrets = Mock()
+
         self.hs = self.setup_test_homeserver()
 
         self.hs.config.registration_shared_secret = "shared"
@@ -82,13 +84,14 @@ class UserRegisterTestCase(unittest.HomeserverTestCase):
         Calling GET on the endpoint will return a randomised nonce, using the
         homeserver's secrets provider.
         """
-        with patch("secrets.token_hex") as token_hex:
-            # Patch secrets.token_hex for the duration of this context
-            token_hex.return_value = "abcd"
+        secrets = Mock()
+        secrets.token_hex = Mock(return_value="abcd")
 
-            channel = self.make_request("GET", self.url)
+        self.hs.get_secrets = Mock(return_value=secrets)
 
-            self.assertEqual(channel.json_body, {"nonce": "abcd"})
+        channel = self.make_request("GET", self.url)
+
+        self.assertEqual(channel.json_body, {"nonce": "abcd"})
 
     def test_expired_nonce(self):
         """

tests/storage/test__base.py

@@ -13,7 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import secrets
 
 from tests import unittest
 
@@ -22,7 +21,7 @@ class UpsertManyTests(unittest.HomeserverTestCase):
     def prepare(self, reactor, clock, hs):
         self.storage = hs.get_datastore()
 
-        self.table_name = "table_" + secrets.token_hex(6)
+        self.table_name = "table_" + hs.get_secrets().token_hex(6)
         self.get_success(
             self.storage.db_pool.runInteraction(
                 "create",

tests/unittest.py

@@ -18,7 +18,6 @@ import hashlib
 import hmac
 import inspect
 import logging
-import secrets
 import time
 from typing import Callable, Dict, Iterable, Optional, Tuple, Type, TypeVar, Union
 from unittest.mock import Mock, patch
@@ -627,6 +626,7 @@ class HomeserverTestCase(TestCase):
             str: The new event's ID.
         """
         event_creator = self.hs.get_event_creation_handler()
+        secrets = self.hs.get_secrets()
         requester = create_requester(user)
 
         event, context = self.get_success(

tests/util/test_glob_to_regex.py

@@ -1,59 +0,0 @@
-# Copyright 2021 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-from synapse.util import glob_to_regex
-
-from tests.unittest import TestCase
-
-
-class GlobToRegexTestCase(TestCase):
-    def test_literal_match(self):
-        """patterns without wildcards should match"""
-        pat = glob_to_regex("foobaz")
-        self.assertTrue(
-            pat.match("FoobaZ"), "patterns should match and be case-insensitive"
-        )
-        self.assertFalse(
-            pat.match("x foobaz"), "pattern should not match at word boundaries"
-        )
-
-    def test_wildcard_match(self):
-        pat = glob_to_regex("f?o*baz")
-
-        self.assertTrue(
-            pat.match("FoobarbaZ"),
-            "* should match string and pattern should be case-insensitive",
-        )
-        self.assertTrue(pat.match("foobaz"), "* should match 0 characters")
-        self.assertFalse(pat.match("fooxaz"), "the character after * must match")
-        self.assertFalse(pat.match("fobbaz"), "? should not match 0 characters")
-        self.assertFalse(pat.match("fiiobaz"), "? should not match 2 characters")
-
-    def test_multi_wildcard(self):
-        """patterns with multiple wildcards in a row should match"""
-        pat = glob_to_regex("**baz")
-        self.assertTrue(pat.match("agsgsbaz"), "** should match any string")
-        self.assertTrue(pat.match("baz"), "** should match the empty string")
-        self.assertEqual(pat.pattern, r"\A.{0,}baz\Z")
-
-        pat = glob_to_regex("*?baz")
-        self.assertTrue(pat.match("agsgsbaz"), "*? should match any string")
-        self.assertTrue(pat.match("abaz"), "*? should match a single char")
-        self.assertFalse(pat.match("baz"), "*? should not match the empty string")
-        self.assertEqual(pat.pattern, r"\A.{1,}baz\Z")
-
-        pat = glob_to_regex("a?*?*?baz")
-        self.assertTrue(pat.match("a g baz"), "?*?*? should match 3 chars")
-        self.assertFalse(pat.match("a..baz"), "?*?*? should not match 2 chars")
-        self.assertTrue(pat.match("a.gg.baz"), "?*?*? should match 4 chars")
-        self.assertEqual(pat.pattern, r"\Aa.{3,}baz\Z")

tox.ini

@@ -21,11 +21,13 @@ deps =
     # installed on that).
     #
     # anyway, make sure that we have a recent enough setuptools.
-    setuptools>=18.5
+    setuptools>=18.5 ; python_version >= '3.6'
+    setuptools>=18.5,<51.0.0 ; python_version < '3.6'
 
     # we also need a semi-recent version of pip, because old ones fail to
     # install the "enum34" dependency of cryptography.
-    pip>=10
+    pip>=10 ; python_version >= '3.6'
+    pip>=10,<21.0 ; python_version < '3.6'
 
 # directories/files we run the linters on.
 # if you update this list, make sure to do the same in scripts-dev/lint.sh
@@ -166,7 +168,8 @@ skip_install = true
 usedevelop = false
 deps =
     coverage
-    pip>=10
+    pip>=10 ; python_version >= '3.6'
+    pip>=10,<21.0 ; python_version < '3.6'
 commands=
     coverage combine
     coverage report