disable background updates

Update changelog
1.46.0
2021-11-04 16:08:48 +00:00 · 2021-11-02 13:25:42 +00:00 · 2021-11-02 13:23:01 +00:00 · 2021-10-27 17:06:32 +01:00 · 2021-10-27 14:48:02 +00:00 · 2021-10-27 14:44:45 +01:00
134 changed files with 1411 additions and 425 deletions
@@ -1,3 +1,80 @@
+Synapse 1.46.0 (2021-11-02)
+===========================
+
+The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))
+
+Bugfixes
+--------
+
+- Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\#11196](https://github.com/matrix-org/synapse/issues/11196))
+
+
+Synapse 1.46.0rc1 (2021-10-27)
+==============================
+
+Features
+--------
+
+- Add support for Ubuntu 21.10 "Impish Indri". ([\#11024](https://github.com/matrix-org/synapse/issues/11024))
+- Port the Password Auth Providers module interface to the new generic interface. ([\#10548](https://github.com/matrix-org/synapse/issues/10548), [\#11180](https://github.com/matrix-org/synapse/issues/11180))
+- Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11088](https://github.com/matrix-org/synapse/issues/11088), [\#11181](https://github.com/matrix-org/synapse/issues/11181), [\#11192](https://github.com/matrix-org/synapse/issues/11192))
+- Users admin API can now also modify user type in addition to allowing it to be set on user creation. ([\#11174](https://github.com/matrix-org/synapse/issues/11174))
+
+
+Bugfixes
+--------
+
+- Newly-created public rooms are now only assigned an alias if the room's creation has not been blocked by permission settings. Contributed by @AndrewFerr. ([\#10930](https://github.com/matrix-org/synapse/issues/10930))
+- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#11001](https://github.com/matrix-org/synapse/issues/11001), [\#11009](https://github.com/matrix-org/synapse/issues/11009), [\#11012](https://github.com/matrix-org/synapse/issues/11012))
+- Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room. ([\#11027](https://github.com/matrix-org/synapse/issues/11027))
+- Fix a bug where setting a user's `external_id` via the admin API returns 500 and deletes user's existing external mappings if that external ID is already mapped. ([\#11051](https://github.com/matrix-org/synapse/issues/11051))
+- Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private. ([\#11075](https://github.com/matrix-org/synapse/issues/11075))
+- Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding. ([\#11077](https://github.com/matrix-org/synapse/issues/11077), [\#11089](https://github.com/matrix-org/synapse/issues/11089))
+- Fix broken export-data admin command and add test script checking the command to CI. ([\#11078](https://github.com/matrix-org/synapse/issues/11078))
+- Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API. ([\#11101](https://github.com/matrix-org/synapse/issues/11101))
+- Fix local users who left all their rooms being removed from the user directory, even if the `search_all_users` config option was enabled. ([\#11103](https://github.com/matrix-org/synapse/issues/11103))
+- Fix a bug which caused the module API's `get_user_ip_and_agents` function to always fail on workers. `get_user_ip_and_agents` was introduced in 1.44.0 and did not function correctly on worker processes at the time. ([\#11112](https://github.com/matrix-org/synapse/issues/11112))
+- Identity server connection is no longer ignoring `ip_range_whitelist`. ([\#11120](https://github.com/matrix-org/synapse/issues/11120))
+- Fix a bug introduced in Synapse 1.45.0 breaking the configuration file parsing script. ([\#11145](https://github.com/matrix-org/synapse/issues/11145))
+- Fix a performance regression introduced in 1.44.0 which could cause client requests to time out when making large numbers of outbound requests. ([\#11177](https://github.com/matrix-org/synapse/issues/11177), [\#11190](https://github.com/matrix-org/synapse/issues/11190))
+- Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch. ([\#10975](https://github.com/matrix-org/synapse/issues/10975))
+
+
+Improved Documentation
+----------------------
+
+- Fix broken links relating to module API deprecation in the upgrade notes. ([\#11069](https://github.com/matrix-org/synapse/issues/11069))
+- Add more information about what happens when a user is deactivated. ([\#11083](https://github.com/matrix-org/synapse/issues/11083))
+- Clarify the the sample log config can be copied from the documentation without issue. ([\#11092](https://github.com/matrix-org/synapse/issues/11092))
+- Update the admin API documentation with an updated list of the characters allowed in registration tokens. ([\#11093](https://github.com/matrix-org/synapse/issues/11093))
+- Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints. ([\#11096](https://github.com/matrix-org/synapse/issues/11096))
+- Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion. ([\#11118](https://github.com/matrix-org/synapse/issues/11118))
+- Document the version of Synapse each module callback was introduced in. ([\#11132](https://github.com/matrix-org/synapse/issues/11132))
+- Document the version of Synapse that introduced each module API method. ([\#11183](https://github.com/matrix-org/synapse/issues/11183))
+
+
+Internal Changes
+----------------
+- Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection. ([\#10984](https://github.com/matrix-org/synapse/issues/10984))
+- Include rejected status when we log events. ([\#11008](https://github.com/matrix-org/synapse/issues/11008))
+- Add some extra logging to the event persistence code. ([\#11014](https://github.com/matrix-org/synapse/issues/11014))
+- Rearrange the internal workings of the incremental user directory updates. ([\#11035](https://github.com/matrix-org/synapse/issues/11035))
+- Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room. ([\#11143](https://github.com/matrix-org/synapse/issues/11143))
+- Add and improve type hints. ([\#10972](https://github.com/matrix-org/synapse/issues/10972), [\#11055](https://github.com/matrix-org/synapse/issues/11055), [\#11066](https://github.com/matrix-org/synapse/issues/11066), [\#11076](https://github.com/matrix-org/synapse/issues/11076), [\#11095](https://github.com/matrix-org/synapse/issues/11095), [\#11109](https://github.com/matrix-org/synapse/issues/11109), [\#11121](https://github.com/matrix-org/synapse/issues/11121), [\#11146](https://github.com/matrix-org/synapse/issues/11146))
+- Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse. ([\#11054](https://github.com/matrix-org/synapse/issues/11054))
+- Remove dead code from `MediaFilePaths`. ([\#11056](https://github.com/matrix-org/synapse/issues/11056))
+- Be more lenient when parsing oEmbed response versions. ([\#11065](https://github.com/matrix-org/synapse/issues/11065))
+- Create a separate module for the retention configuration. ([\#11070](https://github.com/matrix-org/synapse/issues/11070))
+- Clean up some of the federation event authentication code for clarity. ([\#11115](https://github.com/matrix-org/synapse/issues/11115), [\#11116](https://github.com/matrix-org/synapse/issues/11116), [\#11122](https://github.com/matrix-org/synapse/issues/11122))
+- Add docstrings and comments to the application service ephemeral event sending code. ([\#11138](https://github.com/matrix-org/synapse/issues/11138))
+- Update the `sign_json` script to support inline configuration of the signing key. ([\#11139](https://github.com/matrix-org/synapse/issues/11139))
+- Fix broken link in the docker image README. ([\#11144](https://github.com/matrix-org/synapse/issues/11144))
+- Always dump logs from unit tests during CI runs. ([\#11068](https://github.com/matrix-org/synapse/issues/11068))
+- Add tests for `MediaFilePaths` class. ([\#11057](https://github.com/matrix-org/synapse/issues/11057))
+- Simplify the user admin API tests. ([\#11048](https://github.com/matrix-org/synapse/issues/11048))
+- Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s. ([\#11071](https://github.com/matrix-org/synapse/issues/11071))
+
+
 Synapse 1.45.1 (2021-10-20)
 ===========================

@@ -1 +0,0 @@
-Port the Password Auth Providers module interface to the new generic interface.
@@ -1 +0,0 @@
-Add type hints to `synapse.storage.databases.main.client_ips`.
@@ -1 +0,0 @@
-Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch.
@@ -1 +0,0 @@
-Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection.
@@ -1 +0,0 @@
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.
@@ -1 +0,0 @@
-Include rejected status when we log events.
@@ -1 +0,0 @@
-Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.
@@ -1 +0,0 @@
-Add some extra logging to the event persistence code.
@@ -1 +0,0 @@
-Add support for Ubuntu 21.10 "Impish Indri".
@@ -1 +0,0 @@
-Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room.
@@ -1 +0,0 @@
-Rearrange the internal workings of the incremental user directory updates.
@@ -1 +0,0 @@
-Simplify the user admin API tests.
@@ -1 +0,0 @@
-Fix a bug where setting a user's external_id via the admin API returns 500 and deletes users existing external mappings if that external ID is already mapped.
@@ -1 +0,0 @@
-Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse.
@@ -1 +0,0 @@
-Improve type hints for `_wrap_in_base_path` decorator used by `MediaFilePaths`.
@@ -1 +0,0 @@
-Remove dead code from `MediaFilePaths`.
@@ -1 +0,0 @@
-Add tests for `MediaFilePaths` class.
@@ -1 +0,0 @@
-Be more lenient when parsing oEmbed response versions.
@@ -1 +0,0 @@
-Add type hints to `synapse.events`.
@@ -1 +0,0 @@
-Always dump logs from unit tests during CI runs.
@@ -1 +0,0 @@
-Fix broken links relating to module API deprecation in the upgrade notes.
@@ -1 +0,0 @@
-Create a separate module for the retention configuration.
@@ -1 +0,0 @@
-Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s.
@@ -1 +0,0 @@
-Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private.
@@ -1 +0,0 @@
-Fix type hints in the relations tests.
@@ -1 +0,0 @@
-Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.
@@ -1 +0,0 @@
-Fix broken export-data admin command and add test script checking the command to CI.
@@ -1 +0,0 @@
-Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.
@@ -1 +0,0 @@
-Clarify the the sample log config can be copied from the documentation without issue.
@@ -1 +0,0 @@
-Update the admin API documentation with an updated list of the characters allowed in registration tokens.
@@ -1 +0,0 @@
-Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints.
@@ -1 +0,0 @@
-Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API.
@@ -1 +0,0 @@
-Fix local users who left all their rooms being removed from the user directory, even if the "search_all_users" config option was enabled.
@@ -1 +0,0 @@
-Add missing type hints to `synapse.api` module.
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
@@ -1 +0,0 @@
-Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion.
@@ -1 +0,0 @@
-Identity server connection is no longer ignoring `ip_range_whitelist`.
@@ -1 +0,0 @@
-Add type hints for event fetching.
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
@@ -1 +0,0 @@
-Document the version of Synapse each module callback was introduced in.
@@ -1 +0,0 @@
-Add docstrings and comments to the application service ephemeral event sending code.
@@ -1 +0,0 @@
-Update the `sign_json` script to support inline configuration of the signing key.
@@ -1 +0,0 @@
-Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room.
@@ -1 +0,0 @@
-Fix broken link in the docker image README.
@@ -1 +0,0 @@
-Add missing type hints to `synapse.crypto`.
@@ -1,3 +1,19 @@
+matrix-synapse-py3 (1.46.0) stable; urgency=medium
+
+  [ Richard van der Hoff ]
+  * Compress debs with xz, to fix incompatibility of impish debs with reprepro.
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.46.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 02 Nov 2021 13:22:53 +0000
+
+matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.46.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Oct 2021 14:04:04 +0100
+
 matrix-synapse-py3 (1.45.1) stable; urgency=medium

  * New synapse release 1.45.1.
@@ -51,5 +51,11 @@ override_dh_shlibdeps:
 override_dh_virtualenv:
 	./debian/build_virtualenv

+override_dh_builddeb:
+        # force the compression to xzip, to stop dpkg-deb on impish defaulting to zstd
+        # (which requires reprepro 5.3.0-1.3, which is currently only in 'experimental' in Debian:
+        # https://metadata.ftp-master.debian.org/changelogs/main/r/reprepro/reprepro_5.3.0-1.3_changelog)
+	dh_builddeb -- -Zxz
+
 %:
 	dh $@ --with python-virtualenv
@@ -50,7 +50,8 @@ It returns a JSON body like the following:
            "auth_provider": "<provider2>",
            "external_id": "<user_id_provider_2>"
        }
-    ]
+    ],
+    "user_type": null
 }
 ```

@@ -97,7 +98,8 @@ with a body of:
    ],
    "avatar_url": "<avatar_url>",
    "admin": false,
-    "deactivated": false
+    "deactivated": false,
+    "user_type": null
 }
 ```

@@ -135,6 +137,9 @@ Body parameters:
  unchanged on existing accounts and set to `false` for new accounts.
  A user cannot be erased by deactivating with this API. For details on
  deactivating users see [Deactivate Account](#deactivate-account).
+- `user_type` - string or null, optional. If provided, the user type will be
+  adjusted. If `null` given, the user type will be cleared. Other 
+  allowed options are: `bot` and `support`.

 If the user already exists then optional parameters default to the current value.

@@ -341,6 +346,7 @@ The following actions are performed when deactivating an user:
 - Remove all 3PIDs from the homeserver
 - Delete all devices and E2EE keys
 - Delete all access tokens
+- Delete all pushers
 - Delete the password hash
 - Removal from all rooms the user is a member of
 - Remove the user from the user directory
@@ -354,6 +360,15 @@ is set to `true`:
 - Remove the user's avatar URL
 - Mark the user as erased

+The following actions are **NOT** performed. The list may be incomplete.
+
+- Remove mappings of SSO IDs
+- [Delete media uploaded](#delete-media-uploaded-by-a-user) by user (included avatar images)
+- Delete sent and received messages
+- Delete E2E cross-signing keys
+- Remove the user's creation (registration) timestamp
+- [Remove rate limit overrides](#override-ratelimiting-for-users)
+- Remove from monthly active users

 ## Reset password

@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.45.1"
+__version__ = "1.46.0"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -176,6 +176,7 @@ class RelationTypes:
    ANNOTATION = "m.annotation"
    REPLACE = "m.replace"
    REFERENCE = "m.reference"
+    THREAD = "io.element.thread"


 class LimitBlockingTypes:
@@ -31,6 +31,7 @@ import twisted
 from twisted.internet import defer, error, reactor
 from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
+from twisted.python.threadpool import ThreadPool

 import synapse
 from synapse.api.constants import MAX_PDU_SIZE
@@ -48,6 +49,7 @@ from synapse.metrics.background_process_metrics import wrap_as_background_proces
 from synapse.metrics.jemalloc import setup_jemalloc_stats
 from synapse.util.caches.lrucache import setup_expire_lru_cache_entries
 from synapse.util.daemonize import daemonize_process
+from synapse.util.gai_resolver import GAIResolver
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string

@@ -294,7 +296,7 @@ def listen_ssl(
    return r


-def refresh_certificate(hs):
+def refresh_certificate(hs: "HomeServer"):
    """
    Refresh the TLS certificates that Synapse is using by re-reading them from
    disk and updating the TLS context factories to use them.
@@ -338,9 +340,19 @@ async def start(hs: "HomeServer"):
    Args:
        hs: homeserver instance
    """
+    reactor = hs.get_reactor()
+
+    # We want to use a separate thread pool for the resolver so that large
+    # numbers of DNS requests don't starve out other users of the threadpool.
+    resolver_threadpool = ThreadPool(name="gai_resolver")
+    resolver_threadpool.start()
+    reactor.addSystemEventTrigger("during", "shutdown", resolver_threadpool.stop)
+    reactor.installNameResolver(
+        GAIResolver(reactor, getThreadPool=lambda: resolver_threadpool)
+    )
+
    # Set up the SIGHUP machinery.
    if hasattr(signal, "SIGHUP"):
-        reactor = hs.get_reactor()

        @wrap_as_background_process("sighup")
        def handle_sighup(*args, **kwargs):
@@ -419,11 +431,11 @@ async def start(hs: "HomeServer"):
        atexit.register(gc.freeze)


-def setup_sentry(hs):
+def setup_sentry(hs: "HomeServer"):
    """Enable sentry integration, if enabled in configuration

    Args:
-        hs (synapse.server.HomeServer)
+        hs
    """

    if not hs.config.metrics.sentry_enabled:
@@ -449,7 +461,7 @@ def setup_sentry(hs):
        scope.set_tag("worker_name", name)


-def setup_sdnotify(hs):
+def setup_sdnotify(hs: "HomeServer"):
    """Adds process state hooks to tell systemd what we are up to."""

    # Tell systemd our state, if we're using it. This will silently fail if
@@ -68,11 +68,11 @@ class AdminCmdServer(HomeServer):
    DATASTORE_CLASS = AdminCmdSlavedStore


-async def export_data_command(hs, args):
+async def export_data_command(hs: HomeServer, args):
    """Export data for a user.

    Args:
-        hs (HomeServer)
+        hs
        args (argparse.Namespace)
    """

@@ -131,10 +131,10 @@ class KeyUploadServlet(RestServlet):

    PATTERNS = client_patterns("/keys/upload(/(?P<device_id>[^/]+))?$")

-    def __init__(self, hs):
+    def __init__(self, hs: HomeServer):
        """
        Args:
-            hs (synapse.server.HomeServer): server
+            hs: server
        """
        super().__init__()
        self.auth = hs.get_auth()
@@ -412,7 +412,7 @@ def format_config_error(e: ConfigError) -> Iterator[str]:
        e = e.__cause__


-def run(hs):
+def run(hs: HomeServer):
    PROFILE_SYNAPSE = False
    if PROFILE_SYNAPSE:

@@ -15,11 +15,15 @@ import logging
 import math
 import resource
 import sys
+from typing import TYPE_CHECKING

 from prometheus_client import Gauge

 from synapse.metrics.background_process_metrics import wrap_as_background_process

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger("synapse.app.homeserver")

 # Contains the list of processes we will be monitoring
@@ -41,7 +45,7 @@ registered_reserved_users_mau_gauge = Gauge(


@wrap_as_background_process("phone_stats_home")
-async def phone_stats_home(hs, stats, stats_process=_stats_process):
+async def phone_stats_home(hs: "HomeServer", stats, stats_process=_stats_process):
    logger.info("Gathering stats for reporting")
    now = int(hs.get_clock().time())
    uptime = int(now - hs.start_time)
@@ -142,7 +146,7 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
        logger.warning("Error reporting stats: %s", e)


-def start_phone_stats_home(hs):
+def start_phone_stats_home(hs: "HomeServer"):
    """
    Start the background tasks which report phone home stats.
    """
@@ -27,6 +27,7 @@ from synapse.util.caches.response_cache import ResponseCache

 if TYPE_CHECKING:
    from synapse.appservice import ApplicationService
+    from synapse.server import HomeServer

 logger = logging.getLogger(__name__)

@@ -84,7 +85,7 @@ class ApplicationServiceApi(SimpleHttpClient):
    pushing.
    """

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)
        self.clock = hs.get_clock()

@@ -1,4 +1,5 @@
 # Copyright 2015, 2016 OpenMarket Ltd
+# Copyright 2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -11,25 +12,44 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import sys
+
 from synapse.config._base import ConfigError
+from synapse.config.homeserver import HomeServerConfig

-if __name__ == "__main__":
-    import sys

-    from synapse.config.homeserver import HomeServerConfig
+def main(args):
+    action = args[1] if len(args) > 1 and args[1] == "read" else None
+    # If we're reading a key in the config file, then `args[1]` will be `read`  and `args[2]`
+    # will be the key to read.
+    # We'll want to rework this code if we want to support more actions than just `read`.
+    load_config_args = args[3:] if action else args[1:]

-    action = sys.argv[1]
+    try:
+        config = HomeServerConfig.load_config("", load_config_args)
+    except ConfigError as e:
+        sys.stderr.write("\n" + str(e) + "\n")
+        sys.exit(1)
+
+    print("Config parses OK!")

    if action == "read":
-        key = sys.argv[2]
+        key = args[2]
+        key_parts = key.split(".")
+
+        value = config
        try:
-            config = HomeServerConfig.load_config("", sys.argv[3:])
-        except ConfigError as e:
-            sys.stderr.write("\n" + str(e) + "\n")
+            while len(key_parts):
+                value = getattr(value, key_parts[0])
+                key_parts.pop(0)
+
+            print(f"\n{key}: {value}")
+        except AttributeError:
+            print(
+                f"\nNo '{key}' key could be found in the provided configuration file."
+            )
            sys.exit(1)

-        print(getattr(config, key))
-        sys.exit(0)
-    else:
-        sys.stderr.write("Unknown command %r\n" % (action,))
-        sys.exit(1)
+
+if __name__ == "__main__":
+    main(sys.argv)
@@ -26,6 +26,8 @@ class ExperimentalConfig(Config):

        # Whether to enable experimental MSC1849 (aka relations) support
        self.msc1849_enabled = config.get("experimental_msc1849_support_enabled", True)
+        # MSC3440 (thread relation)
+        self.msc3440_enabled: bool = experimental.get("msc3440_enabled", False)

        # MSC3026 (busy presence state)
        self.msc3026_enabled: bool = experimental.get("msc3026_enabled", False)
@@ -18,6 +18,7 @@ import os
 import sys
 import threading
 from string import Template
+from typing import TYPE_CHECKING

 import yaml
 from zope.interface import implementer
@@ -38,6 +39,9 @@ from synapse.util.versionstring import get_version_string

 from ._base import Config, ConfigError

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 DEFAULT_LOG_CONFIG = Template(
    """\
 # Log configuration for Synapse.
@@ -306,7 +310,10 @@ def _reload_logging_config(log_config_path):


 def setup_logging(
-    hs, config, use_worker_options=False, logBeginner: LogBeginner = globalLogBeginner
+    hs: "HomeServer",
+    config,
+    use_worker_options=False,
+    logBeginner: LogBeginner = globalLogBeginner,
 ) -> None:
    """
    Set up the logging subsystem.
@@ -386,6 +386,7 @@ class EventClientSerializer:
    def __init__(self, hs: "HomeServer"):
        self.store = hs.get_datastore()
        self._msc1849_enabled = hs.config.experimental.msc1849_enabled
+        self._msc3440_enabled = hs.config.experimental.msc3440_enabled

    async def serialize_event(
        self,
@@ -462,6 +463,22 @@ class EventClientSerializer:
                    "sender": edit.sender,
                }

+            # If this event is the start of a thread, include a summary of the replies.
+            if self._msc3440_enabled:
+                (
+                    thread_count,
+                    latest_thread_event,
+                ) = await self.store.get_thread_summary(event_id)
+                if latest_thread_event:
+                    r = serialized_event["unsigned"].setdefault("m.relations", {})
+                    r[RelationTypes.THREAD] = {
+                        # Don't bundle aggregations as this could recurse forever.
+                        "latest_event": await self.serialize_event(
+                            latest_thread_event, time_now, bundle_aggregations=False
+                        ),
+                        "count": thread_count,
+                    }
+
        return serialized_event

    async def serialize_events(
@@ -14,6 +14,7 @@
 # limitations under the License.
 import logging
 from collections import namedtuple
+from typing import TYPE_CHECKING

 from synapse.api.constants import MAX_DEPTH, EventContentFields, EventTypes, Membership
 from synapse.api.errors import Codes, SynapseError
@@ -25,11 +26,15 @@ from synapse.events.utils import prune_event, validate_canonicaljson
 from synapse.http.servlet import assert_params_in_dict
 from synapse.types import JsonDict, get_domain_from_id

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
+
 logger = logging.getLogger(__name__)


 class FederationBase:
-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.hs = hs

        self.server_name = hs.hostname
@@ -467,7 +467,7 @@ class FederationServer(FederationBase):

    async def on_room_state_request(
        self, origin: str, room_id: str, event_id: Optional[str]
-    ) -> Tuple[int, Dict[str, Any]]:
+    ) -> Tuple[int, JsonDict]:
        origin_host, _ = parse_server_name(origin)
        await self.check_server_matches_acl(origin_host, room_id)

@@ -481,7 +481,7 @@ class FederationServer(FederationBase):
        # - but that's non-trivial to get right, and anyway somewhat defeats
        # the point of the linearizer.
        with (await self._server_linearizer.queue((origin, room_id))):
-            resp = dict(
+            resp: JsonDict = dict(
                await self._state_resp_cache.wrap(
                    (room_id, event_id),
                    self._on_context_state_request_compute,
@@ -1061,11 +1061,12 @@ class FederationServer(FederationBase):

                origin, event = next

-            lock = await self.store.try_acquire_lock(
+            new_lock = await self.store.try_acquire_lock(
                _INBOUND_EVENT_HANDLING_LOCK_NAME, room_id
            )
-            if not lock:
+            if not new_lock:
                return
+            lock = new_lock

    def __str__(self) -> str:
        return "<ReplicationLayer(%s)>" % self.server_name
@@ -62,7 +62,6 @@ from synapse.http.server import finish_request, respond_with_html
 from synapse.http.site import SynapseRequest
 from synapse.logging.context import defer_to_thread
 from synapse.metrics.background_process_metrics import run_as_background_process
-from synapse.module_api import ModuleApi
 from synapse.storage.roommember import ProfileInfo
 from synapse.types import JsonDict, Requester, UserID
 from synapse.util import stringutils as stringutils
@@ -73,6 +72,7 @@ from synapse.util.stringutils import base62_encode
 from synapse.util.threepids import canonicalise_email

 if TYPE_CHECKING:
+    from synapse.module_api import ModuleApi
    from synapse.rest.client.login import LoginResponse
    from synapse.server import HomeServer

@@ -1818,7 +1818,9 @@ def load_legacy_password_auth_providers(hs: "HomeServer") -> None:


 def load_single_legacy_password_auth_provider(
-    module: Type, config: JsonDict, api: ModuleApi
+    module: Type,
+    config: JsonDict,
+    api: "ModuleApi",
 ) -> None:
    try:
        provider = module(config=config, account_handler=api)
@@ -145,7 +145,7 @@ class DirectoryHandler:
            if not self.config.roomdirectory.is_alias_creation_allowed(
                user_id, room_id, room_alias_str
            ):
-                # Lets just return a generic message, as there may be all sorts of
+                # Let's just return a generic message, as there may be all sorts of
                # reasons why we said no. TODO: Allow configurable error messages
                # per alias creation rule?
                raise SynapseError(403, "Not allowed to create alias")
@@ -461,7 +461,7 @@ class DirectoryHandler:
            if not self.config.roomdirectory.is_publishing_room_allowed(
                user_id, room_id, room_aliases
            ):
-                # Lets just return a generic message, as there may be all sorts of
+                # Let's just return a generic message, as there may be all sorts of
                # reasons why we said no. TODO: Allow configurable error messages
                # per alias creation rule?
                raise SynapseError(403, "Not allowed to publish room")
@@ -361,6 +361,7 @@ class FederationEventHandler:
        # need to.
        await self._event_creation_handler.cache_joined_hosts_for_event(event, context)

+        await self._check_for_soft_fail(event, None, origin=origin)
        await self._run_push_actions_and_persist_event(event, context)
        return event, context

@@ -402,29 +403,28 @@ class FederationEventHandler:
        """Persists the events returned by a send_join

        Checks the auth chain is valid (and passes auth checks) for the
-        state and event. Then persists the auth chain and state atomically.
-        Persists the event separately. Notifies about the persisted events
-        where appropriate.
-
-        Will attempt to fetch missing auth events.
+        state and event. Then persists all of the events.
+        Notifies about the persisted events where appropriate.

        Args:
            origin: Where the events came from
-            room_id,
+            room_id:
            auth_events
            state
            event
            room_version: The room version we expect this room to have, and
                will raise if it doesn't match the version in the create event.
+
+        Returns:
+            The stream ID after which all events have been persisted.
+
+        Raises:
+            SynapseError if the response is in some way invalid.
        """
-        events_to_context = {}
        for e in itertools.chain(auth_events, state):
            e.internal_metadata.outlier = True
-            events_to_context[e.event_id] = EventContext.for_outlier()

-        event_map = {
-            e.event_id: e for e in itertools.chain(auth_events, state, [event])
-        }
+        event_map = {e.event_id: e for e in itertools.chain(auth_events, state)}

        create_event = None
        for e in auth_events:
@@ -444,64 +444,36 @@ class FederationEventHandler:
        if room_version.identifier != room_version_id:
            raise SynapseError(400, "Room version mismatch")

-        missing_auth_events = set()
-        for e in itertools.chain(auth_events, state, [event]):
-            for e_id in e.auth_event_ids():
-                if e_id not in event_map:
-                    missing_auth_events.add(e_id)
+        # filter out any events we have already seen
+        seen_remotes = await self._store.have_seen_events(room_id, event_map.keys())
+        for s in seen_remotes:
+            event_map.pop(s, None)

-        for e_id in missing_auth_events:
-            m_ev = await self._federation_client.get_pdu(
-                [origin],
-                e_id,
-                room_version=room_version,
-                outlier=True,
-                timeout=10000,
-            )
-            if m_ev and m_ev.event_id == e_id:
-                event_map[e_id] = m_ev
-            else:
-                logger.info("Failed to find auth event %r", e_id)
+        # persist the auth chain and state events.
+        #
+        # any invalid events here will be marked as rejected, and we'll carry on.
+        #
+        # any events whose auth events are missing (ie, not in the send_join response,
+        # and not already in our db) will just be ignored. This is correct behaviour,
+        # because the reason that auth_events are missing might be due to us being
+        # unable to validate their signatures. The fact that we can't validate their
+        # signatures right now doesn't mean that we will *never* be able to, so it
+        # is premature to reject them.
+        #
+        await self._auth_and_persist_outliers(room_id, event_map.values())

-        for e in itertools.chain(auth_events, state, [event]):
-            auth_for_e = [
-                event_map[e_id] for e_id in e.auth_event_ids() if e_id in event_map
-            ]
-            if create_event:
-                auth_for_e.append(create_event)
-
-            try:
-                validate_event_for_room_version(room_version, e)
-                check_auth_rules_for_event(room_version, e, auth_for_e)
-            except SynapseError as err:
-                # we may get SynapseErrors here as well as AuthErrors. For
-                # instance, there are a couple of (ancient) events in some
-                # rooms whose senders do not have the correct sigil; these
-                # cause SynapseErrors in auth.check. We don't want to give up
-                # the attempt to federate altogether in such cases.
-
-                logger.warning("Rejecting %s because %s", e.event_id, err.msg)
-
-                if e == event:
-                    raise
-                events_to_context[e.event_id].rejected = RejectedReason.AUTH_ERROR
-
-        if auth_events or state:
-            await self.persist_events_and_notify(
-                room_id,
-                [
-                    (e, events_to_context[e.event_id])
-                    for e in itertools.chain(auth_events, state)
-                ],
+        # and now persist the join event itself.
+        logger.info("Peristing join-via-remote %s", event)
+        with nested_logging_context(suffix=event.event_id):
+            context = await self._state_handler.compute_event_context(
+                event, old_state=state
            )

-        new_event_context = await self._state_handler.compute_event_context(
-            event, old_state=state
-        )
+            context = await self._check_event_auth(origin, event, context)
+            if context.rejected:
+                raise SynapseError(400, "Join event was rejected")

-        return await self.persist_events_and_notify(
-            room_id, [(event, new_event_context)]
-        )
+            return await self.persist_events_and_notify(room_id, [(event, context)])

    @log_function
    async def backfill(
@@ -974,9 +946,15 @@ class FederationEventHandler:
    ) -> None:
        """Called when we have a new non-outlier event.

-        This is called when we have a new event to add to the room DAG - either directly
-        via a /send request, retrieved via get_missing_events after a /send request, or
-        backfilled after a client request.
+        This is called when we have a new event to add to the room DAG. This can be
+        due to:
+           * events received directly via a /send request
+           * events retrieved via get_missing_events after a /send request
+           * events backfilled after a client request.
+
+        It's not currently used for events received from incoming send_{join,knock,leave}
+        requests (which go via on_send_membership_event), nor for joins created by a
+        remote join dance (which go via process_remote_join).

        We need to do auth checks and put it through the StateHandler.

@@ -1012,11 +990,19 @@ class FederationEventHandler:
            logger.exception("Unexpected AuthError from _check_event_auth")
            raise FederationError("ERROR", e.code, e.msg, affected=event.event_id)

+        if not backfilled and not context.rejected:
+            # For new (non-backfilled and non-outlier) events we check if the event
+            # passes auth based on the current state. If it doesn't then we
+            # "soft-fail" the event.
+            await self._check_for_soft_fail(event, state, origin=origin)
+
        await self._run_push_actions_and_persist_event(event, context, backfilled)

-        if backfilled:
+        if backfilled or context.rejected:
            return

+        await self._maybe_kick_guest_users(event)
+
        # For encrypted messages we check that we know about the sending device,
        # if we don't then we mark the device cache for that user as stale.
        if event.type == EventTypes.Encrypted:
@@ -1317,14 +1303,14 @@ class FederationEventHandler:
                for auth_event_id in event.auth_event_ids():
                    ae = persisted_events.get(auth_event_id)
                    if not ae:
-                        logger.warning(
-                            "Event %s relies on auth_event %s, which could not be found.",
-                            event,
-                            auth_event_id,
-                        )
                        # the fact we can't find the auth event doesn't mean it doesn't
                        # exist, which means it is premature to reject `event`. Instead we
                        # just ignore it for now.
+                        logger.warning(
+                            "Dropping event %s, which relies on auth_event %s, which could not be found",
+                            event,
+                            auth_event_id,
+                        )
                        return None
                    auth.append(ae)

@@ -1447,10 +1433,6 @@ class FederationEventHandler:
        except AuthError as e:
            logger.warning("Failed auth resolution for %r because %s", event, e)
            context.rejected = RejectedReason.AUTH_ERROR
-            return context
-
-        await self._check_for_soft_fail(event, state, backfilled, origin=origin)
-        await self._maybe_kick_guest_users(event)

        return context

@@ -1470,7 +1452,6 @@ class FederationEventHandler:
        self,
        event: EventBase,
        state: Optional[Iterable[EventBase]],
-        backfilled: bool,
        origin: str,
    ) -> None:
        """Checks if we should soft fail the event; if so, marks the event as
@@ -1479,15 +1460,8 @@ class FederationEventHandler:
        Args:
            event
            state: The state at the event if we don't have all the event's prev events
-            backfilled: Whether the event is from backfill
            origin: The host the event originates from.
        """
-        # For new (non-backfilled and non-outlier) events we check if the event
-        # passes auth based on the current state. If it doesn't then we
-        # "soft-fail" the event.
-        if backfilled or event.internal_metadata.is_outlier():
-            return
-
        extrem_ids_list = await self._store.get_latest_event_ids_in_room(event.room_id)
        extrem_ids = set(extrem_ids_list)
        prev_event_ids = set(event.prev_event_ids())
@@ -773,6 +773,15 @@ class RoomCreationHandler:
        if not allowed_by_third_party_rules:
            raise SynapseError(403, "Room visibility value not allowed.")

+        if is_public:
+            if not self.config.roomdirectory.is_publishing_room_allowed(
+                user_id, room_id, room_alias
+            ):
+                # Let's just return a generic message, as there may be all sorts of
+                # reasons why we said no. TODO: Allow configurable error messages
+                # per alias creation rule?
+                raise SynapseError(403, "Not allowed to publish room")
+
        directory_handler = self.hs.get_directory_handler()
        if room_alias:
            await directory_handler.create_association(
@@ -783,15 +792,6 @@ class RoomCreationHandler:
                check_membership=False,
            )

-        if is_public:
-            if not self.config.roomdirectory.is_publishing_room_allowed(
-                user_id, room_id, room_alias
-            ):
-                # Lets just return a generic message, as there may be all sorts of
-                # reasons why we said no. TODO: Allow configurable error messages
-                # per alias creation rule?
-                raise SynapseError(403, "Not allowed to publish room")
-
        preset_config = config.get(
            "preset",
            RoomCreationPreset.PRIVATE_CHAT
@@ -21,6 +21,7 @@ import typing
 import urllib.parse
 from io import BytesIO, StringIO
 from typing import (
+    TYPE_CHECKING,
    Callable,
    Dict,
    Generic,
@@ -73,6 +74,9 @@ from synapse.util import json_decoder
 from synapse.util.async_helpers import timeout_deferred
 from synapse.util.metrics import Measure

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)

 outgoing_requests_counter = Counter(
@@ -319,7 +323,7 @@ class MatrixFederationHttpClient:
            requests.
    """

-    def __init__(self, hs, tls_client_options_factory):
+    def __init__(self, hs: "HomeServer", tls_client_options_factory):
        self.hs = hs
        self.signing_key = hs.signing_key
        self.server_name = hs.hostname
@@ -711,7 +715,7 @@ class MatrixFederationHttpClient:
        Returns:
            A list of headers to be added as "Authorization:" headers
        """
-        request = {
+        request: JsonDict = {
            "method": method.decode("ascii"),
            "uri": url_bytes.decode("ascii"),
            "origin": self.server_name,
@@ -22,6 +22,7 @@ import urllib
 from http import HTTPStatus
 from inspect import isawaitable
 from typing import (
+    TYPE_CHECKING,
    Any,
    Awaitable,
    Callable,
@@ -61,6 +62,9 @@ from synapse.util import json_encoder
 from synapse.util.caches import intern_dict
 from synapse.util.iterutils import chunk_seq

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)

 HTML_ERROR_TEMPLATE = """<!DOCTYPE html>
@@ -343,6 +347,11 @@ class DirectServeJsonResource(_AsyncResource):
        return_json_error(f, request)


+_PathEntry = collections.namedtuple(
+    "_PathEntry", ["pattern", "callback", "servlet_classname"]
+)
+
+
 class JsonResource(DirectServeJsonResource):
    """This implements the HttpServer interface and provides JSON support for
    Resources.
@@ -359,14 +368,10 @@ class JsonResource(DirectServeJsonResource):

    isLeaf = True

-    _PathEntry = collections.namedtuple(
-        "_PathEntry", ["pattern", "callback", "servlet_classname"]
-    )
-
-    def __init__(self, hs, canonical_json=True, extract_context=False):
+    def __init__(self, hs: "HomeServer", canonical_json=True, extract_context=False):
        super().__init__(canonical_json, extract_context)
        self.clock = hs.get_clock()
-        self.path_regexs = {}
+        self.path_regexs: Dict[bytes, List[_PathEntry]] = {}
        self.hs = hs

    def register_paths(self, method, path_patterns, callback, servlet_classname):
@@ -391,7 +396,7 @@ class JsonResource(DirectServeJsonResource):
        for path_pattern in path_patterns:
            logger.debug("Registering for %s %s", method, path_pattern.pattern)
            self.path_regexs.setdefault(method, []).append(
-                self._PathEntry(path_pattern, callback, servlet_classname)
+                _PathEntry(path_pattern, callback, servlet_classname)
            )

    def _get_handler_for_request(
@@ -46,6 +46,7 @@ from synapse.http.site import SynapseRequest
 from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.rest.client.login import LoginResponse
+from synapse.storage import DataStore
 from synapse.storage.database import DatabasePool, LoggingTransaction
 from synapse.storage.databases.main.roommember import ProfileInfo
 from synapse.storage.state import StateFilter
@@ -61,6 +62,7 @@ from synapse.util import Clock
 from synapse.util.caches.descriptors import cached

 if TYPE_CHECKING:
+    from synapse.app.generic_worker import GenericWorkerSlavedStore
    from synapse.server import HomeServer

 """
@@ -111,7 +113,9 @@ class ModuleApi:
    def __init__(self, hs: "HomeServer", auth_handler):
        self._hs = hs

-        self._store = hs.get_datastore()
+        # TODO: Fix this type hint once the types for the data stores have been ironed
+        #       out.
+        self._store: Union[DataStore, "GenericWorkerSlavedStore"] = hs.get_datastore()
        self._auth = hs.get_auth()
        self._auth_handler = auth_handler
        self._server_name = hs.hostname
@@ -150,27 +154,42 @@ class ModuleApi:

    @property
    def register_spam_checker_callbacks(self):
-        """Registers callbacks for spam checking capabilities."""
+        """Registers callbacks for spam checking capabilities.
+
+        Added in Synapse v1.37.0.
+        """
        return self._spam_checker.register_callbacks

    @property
    def register_account_validity_callbacks(self):
-        """Registers callbacks for account validity capabilities."""
+        """Registers callbacks for account validity capabilities.
+
+        Added in Synapse v1.39.0.
+        """
        return self._account_validity_handler.register_account_validity_callbacks

    @property
    def register_third_party_rules_callbacks(self):
-        """Registers callbacks for third party event rules capabilities."""
+        """Registers callbacks for third party event rules capabilities.
+
+        Added in Synapse v1.39.0.
+        """
        return self._third_party_event_rules.register_third_party_rules_callbacks

    @property
    def register_presence_router_callbacks(self):
-        """Registers callbacks for presence router capabilities."""
+        """Registers callbacks for presence router capabilities.
+
+        Added in Synapse v1.42.0.
+        """
        return self._presence_router.register_presence_router_callbacks

    @property
    def register_password_auth_provider_callbacks(self):
-        """Registers callbacks for password auth provider capabilities."""
+        """Registers callbacks for password auth provider capabilities.
+
+        Added in Synapse v1.46.0.
+        """
        return self._password_auth_provider.register_password_auth_provider_callbacks

    def register_web_resource(self, path: str, resource: IResource):
@@ -181,6 +200,8 @@ class ModuleApi:
        If multiple modules register a resource for the same path, the module that
        appears the highest in the configuration file takes priority.

+        Added in Synapse v1.37.0.
+
        Args:
            path: The path to register the resource for.
            resource: The resource to attach to this path.
@@ -195,6 +216,8 @@ class ModuleApi:
        """Allows making outbound HTTP requests to remote resources.

        An instance of synapse.http.client.SimpleHttpClient
+
+        Added in Synapse v1.22.0.
        """
        return self._http_client

@@ -204,22 +227,32 @@ class ModuleApi:
        public room list.

        An instance of synapse.module_api.PublicRoomListManager
+
+        Added in Synapse v1.22.0.
        """
        return self._public_room_list_manager

    @property
    def public_baseurl(self) -> str:
-        """The configured public base URL for this homeserver."""
+        """The configured public base URL for this homeserver.
+
+        Added in Synapse v1.39.0.
+        """
        return self._hs.config.server.public_baseurl

    @property
    def email_app_name(self) -> str:
-        """The application name configured in the homeserver's configuration."""
+        """The application name configured in the homeserver's configuration.
+
+        Added in Synapse v1.39.0.
+        """
        return self._hs.config.email.email_app_name

    async def get_userinfo_by_id(self, user_id: str) -> Optional[UserInfo]:
        """Get user info by user_id

+        Added in Synapse v1.41.0.
+
        Args:
            user_id: Fully qualified user id.
        Returns:
@@ -235,6 +268,8 @@ class ModuleApi:
    ) -> Requester:
        """Check the access_token provided for a request

+        Added in Synapse v1.39.0.
+
        Args:
            req: Incoming HTTP request
            allow_guest: True if guest users should be allowed. If this
@@ -260,6 +295,8 @@ class ModuleApi:
    async def is_user_admin(self, user_id: str) -> bool:
        """Checks if a user is a server admin.

+        Added in Synapse v1.39.0.
+
        Args:
            user_id: The Matrix ID of the user to check.

@@ -274,6 +311,8 @@ class ModuleApi:
        Takes a user id provided by the user and adds the @ and :domain to
        qualify it, if necessary

+        Added in Synapse v0.25.0.
+
        Args:
            username (str): provided user id

@@ -287,6 +326,8 @@ class ModuleApi:
    async def get_profile_for_user(self, localpart: str) -> ProfileInfo:
        """Look up the profile info for the user with the given localpart.

+        Added in Synapse v1.39.0.
+
        Args:
            localpart: The localpart to look up profile information for.

@@ -299,6 +340,8 @@ class ModuleApi:
        """Look up the threepids (email addresses and phone numbers) associated with the
        given Matrix user ID.

+        Added in Synapse v1.39.0.
+
        Args:
            user_id: The Matrix user ID to look up threepids for.

@@ -313,6 +356,8 @@ class ModuleApi:
    def check_user_exists(self, user_id):
        """Check if user exists.

+        Added in Synapse v0.25.0.
+
        Args:
            user_id (str): Complete @user:id

@@ -332,6 +377,8 @@ class ModuleApi:
        return that device to the user. Prefer separate calls to register_user and
        register_device.

+        Added in Synapse v0.25.0.
+
        Args:
            localpart (str): The localpart of the new user.
            displayname (str|None): The displayname of the new user.
@@ -352,6 +399,8 @@ class ModuleApi:
    ):
        """Registers a new user with given localpart and optional displayname, emails.

+        Added in Synapse v1.2.0.
+
        Args:
            localpart (str): The localpart of the new user.
            displayname (str|None): The displayname of the new user.
@@ -375,6 +424,8 @@ class ModuleApi:
    def register_device(self, user_id, device_id=None, initial_display_name=None):
        """Register a device for a user and generate an access token.

+        Added in Synapse v1.2.0.
+
        Args:
            user_id (str): full canonical @user:id
            device_id (str|None): The device ID to check, or None to generate
@@ -398,6 +449,8 @@ class ModuleApi:
    ) -> defer.Deferred:
        """Record a mapping from an external user id to a mxid

+        Added in Synapse v1.9.0.
+
        Args:
            auth_provider: identifier for the remote auth provider
            external_id: id on that system
@@ -417,6 +470,8 @@ class ModuleApi:
    ) -> str:
        """Generate a login token suitable for m.login.token authentication

+        Added in Synapse v1.9.0.
+
        Args:
            user_id: gives the ID of the user that the token is for

@@ -436,6 +491,8 @@ class ModuleApi:
    def invalidate_access_token(self, access_token):
        """Invalidate an access token for a user

+        Added in Synapse v0.25.0.
+
        Args:
            access_token(str): access token

@@ -466,6 +523,8 @@ class ModuleApi:
    def run_db_interaction(self, desc, func, *args, **kwargs):
        """Run a function with a database connection

+        Added in Synapse v0.25.0.
+
        Args:
            desc (str): description for the transaction, for metrics etc
            func (func): function to be run. Passed a database cursor object
@@ -489,6 +548,8 @@ class ModuleApi:

        This is deprecated in favor of complete_sso_login_async.

+        Added in Synapse v1.11.1.
+
        Args:
            registered_user_id: The MXID that has been registered as a previous step of
                of this SSO login.
@@ -515,6 +576,8 @@ class ModuleApi:
        want their access token sent to `client_redirect_url`, or redirect them to that
        URL with a token directly if the URL matches with one of the whitelisted clients.

+        Added in Synapse v1.13.0.
+
        Args:
            registered_user_id: The MXID that has been registered as a previous step of
                of this SSO login.
@@ -543,6 +606,8 @@ class ModuleApi:
        (This is exposed for compatibility with the old SpamCheckerApi. We should
        probably deprecate it and replace it with an async method in a subclass.)

+        Added in Synapse v1.22.0.
+
        Args:
            room_id: The room ID to get state events in.
            types: The event type and state key (using None
@@ -563,6 +628,8 @@ class ModuleApi:
    async def create_and_send_event_into_room(self, event_dict: JsonDict) -> EventBase:
        """Create and send an event into a room. Membership events are currently not supported.

+        Added in Synapse v1.22.0.
+
        Args:
            event_dict: A dictionary representing the event to send.
                Required keys are `type`, `room_id`, `sender` and `content`.
@@ -603,6 +670,8 @@ class ModuleApi:

        Note that this method can only be run on the process that is configured to write to the
        presence stream. By default this is the main process.
+
+        Added in Synapse v1.32.0.
        """
        if self._hs._instance_name not in self._hs.config.worker.writers.presence:
            raise Exception(
@@ -657,6 +726,8 @@ class ModuleApi:

        Waits `msec` initially before calling `f` for the first time.

+        Added in Synapse v1.39.0.
+
        Args:
            f: The function to call repeatedly. f can be either synchronous or
                asynchronous, and must follow Synapse's logcontext rules.
@@ -696,6 +767,8 @@ class ModuleApi:
    ):
        """Send an email on behalf of the homeserver.

+        Added in Synapse v1.39.0.
+
        Args:
            recipient: The email address for the recipient.
            subject: The email's subject.
@@ -719,6 +792,8 @@ class ModuleApi:
        By default, Synapse will look for these templates in its configured template
        directory, but another directory to search in can be provided.

+        Added in Synapse v1.39.0.
+
        Args:
            filenames: The name of the template files to look for.
            custom_template_directory: An additional directory to look for the files in.
@@ -736,13 +811,13 @@ class ModuleApi:
        """
        Checks whether an ID (user id, room, ...) comes from this homeserver.

+        Added in Synapse v1.44.0.
+
        Args:
            id: any Matrix id (e.g. user id, room id, ...), either as a raw id,
                e.g. string "@user:example.com" or as a parsed UserID, RoomID, ...
        Returns:
            True if id comes from this homeserver, False otherwise.
-
-        Added in Synapse v1.44.0.
        """
        if isinstance(id, DomainSpecificString):
            return self._hs.is_mine(id)
@@ -755,6 +830,8 @@ class ModuleApi:
        """
        Return the list of user IPs and agents for a user.

+        Added in Synapse v1.44.0.
+
        Args:
            user_id: the id of a user, local or remote
            since_ts: a timestamp in seconds since the epoch,
@@ -763,8 +840,6 @@ class ModuleApi:
            The list of all UserIpAndAgent that the user has
            used to connect to this homeserver since `since_ts`.
            If the user is remote, this list is empty.
-
-        Added in Synapse v1.44.0.
        """
        # Don't hit the db if this is not a local user.
        is_mine = False
@@ -803,6 +878,8 @@ class PublicRoomListManager:
    async def room_is_in_public_room_list(self, room_id: str) -> bool:
        """Checks whether a room is in the public room list.

+        Added in Synapse v1.22.0.
+
        Args:
            room_id: The ID of the room.

@@ -819,6 +896,8 @@ class PublicRoomListManager:
    async def add_room_to_public_room_list(self, room_id: str) -> None:
        """Publishes a room to the public room list.

+        Added in Synapse v1.22.0.
+
        Args:
            room_id: The ID of the room.
        """
@@ -827,6 +906,8 @@ class PublicRoomListManager:
    async def remove_room_from_public_room_list(self, room_id: str) -> None:
        """Removes a room from the public room list.

+        Added in Synapse v1.22.0.
+
        Args:
            room_id: The ID of the room.
        """
@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import TYPE_CHECKING
+
 from synapse.http.server import JsonResource
 from synapse.replication.http import (
    account_data,
@@ -26,16 +28,19 @@ from synapse.replication.http import (
    streams,
 )

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 REPLICATION_PREFIX = "/_synapse/replication"


 class ReplicationRestResource(JsonResource):
-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        # We enable extracting jaeger contexts here as these are internal APIs.
        super().__init__(hs, canonical_json=False, extract_context=True)
        self.register_servlets(hs)

-    def register_servlets(self, hs):
+    def register_servlets(self, hs: "HomeServer"):
        send_event.register_servlets(hs, self)
        federation.register_servlets(hs, self)
        presence.register_servlets(hs, self)
@@ -17,7 +17,7 @@ import logging
 import re
 import urllib
 from inspect import signature
-from typing import TYPE_CHECKING, Dict, List, Tuple
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, List, Tuple

 from prometheus_client import Counter, Gauge

@@ -156,7 +156,7 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
        pass

    @classmethod
-    def make_client(cls, hs):
+    def make_client(cls, hs: "HomeServer"):
        """Create a client that makes requests.

        Returns a callable that accepts the same parameters as
@@ -208,7 +208,9 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
                    url_args.append(txn_id)

                if cls.METHOD == "POST":
-                    request_func = client.post_json_get_json
+                    request_func: Callable[
+                        ..., Awaitable[Any]
+                    ] = client.post_json_get_json
                elif cls.METHOD == "PUT":
                    request_func = client.put_json
                elif cls.METHOD == "GET":
@@ -13,10 +13,14 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.http.servlet import parse_json_object_from_request
 from synapse.replication.http._base import ReplicationEndpoint

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -37,7 +41,7 @@ class ReplicationUserAccountDataRestServlet(ReplicationEndpoint):
    PATH_ARGS = ("user_id", "account_data_type")
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.handler = hs.get_account_data_handler()
@@ -78,7 +82,7 @@ class ReplicationRoomAccountDataRestServlet(ReplicationEndpoint):
    PATH_ARGS = ("user_id", "room_id", "account_data_type")
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.handler = hs.get_account_data_handler()
@@ -119,7 +123,7 @@ class ReplicationAddTagRestServlet(ReplicationEndpoint):
    PATH_ARGS = ("user_id", "room_id", "tag")
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.handler = hs.get_account_data_handler()
@@ -162,7 +166,7 @@ class ReplicationRemoveTagRestServlet(ReplicationEndpoint):
    )
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.handler = hs.get_account_data_handler()
@@ -183,7 +187,7 @@ class ReplicationRemoveTagRestServlet(ReplicationEndpoint):
        return 200, {"max_stream_id": max_stream_id}


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationUserAccountDataRestServlet(hs).register(http_server)
    ReplicationRoomAccountDataRestServlet(hs).register(http_server)
    ReplicationAddTagRestServlet(hs).register(http_server)
@@ -13,9 +13,13 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.replication.http._base import ReplicationEndpoint

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -51,7 +55,7 @@ class ReplicationUserDevicesResyncRestServlet(ReplicationEndpoint):
    PATH_ARGS = ("user_id",)
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.device_list_updater = hs.get_device_handler().device_list_updater
@@ -68,5 +72,5 @@ class ReplicationUserDevicesResyncRestServlet(ReplicationEndpoint):
        return 200, user_devices


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationUserDevicesResyncRestServlet(hs).register(http_server)
@@ -13,6 +13,7 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import make_event_from_dict
@@ -21,6 +22,9 @@ from synapse.http.servlet import parse_json_object_from_request
 from synapse.replication.http._base import ReplicationEndpoint
 from synapse.util.metrics import Measure

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -56,7 +60,7 @@ class ReplicationFederationSendEventsRestServlet(ReplicationEndpoint):
    NAME = "fed_send_events"
    PATH_ARGS = ()

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.store = hs.get_datastore()
@@ -151,7 +155,7 @@ class ReplicationFederationSendEduRestServlet(ReplicationEndpoint):
    NAME = "fed_send_edu"
    PATH_ARGS = ("edu_type",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.store = hs.get_datastore()
@@ -194,7 +198,7 @@ class ReplicationGetQueryRestServlet(ReplicationEndpoint):
    # This is a query, so let's not bother caching
    CACHE = False

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.store = hs.get_datastore()
@@ -238,7 +242,7 @@ class ReplicationCleanRoomRestServlet(ReplicationEndpoint):
    NAME = "fed_cleanup_room"
    PATH_ARGS = ("room_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.store = hs.get_datastore()
@@ -273,7 +277,7 @@ class ReplicationStoreRoomOnOutlierMembershipRestServlet(ReplicationEndpoint):
    NAME = "store_room_on_outlier_membership"
    PATH_ARGS = ("room_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.store = hs.get_datastore()
@@ -289,7 +293,7 @@ class ReplicationStoreRoomOnOutlierMembershipRestServlet(ReplicationEndpoint):
        return 200, {}


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationFederationSendEventsRestServlet(hs).register(http_server)
    ReplicationFederationSendEduRestServlet(hs).register(http_server)
    ReplicationGetQueryRestServlet(hs).register(http_server)
@@ -13,10 +13,14 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.http.servlet import parse_json_object_from_request
 from synapse.replication.http._base import ReplicationEndpoint

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -30,7 +34,7 @@ class RegisterDeviceReplicationServlet(ReplicationEndpoint):
    NAME = "device_check_registered"
    PATH_ARGS = ("user_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)
        self.registration_handler = hs.get_registration_handler()

@@ -82,5 +86,5 @@ class RegisterDeviceReplicationServlet(ReplicationEndpoint):
        return 200, res


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    RegisterDeviceReplicationServlet(hs).register(http_server)
@@ -45,7 +45,7 @@ class ReplicationRemoteJoinRestServlet(ReplicationEndpoint):
    NAME = "remote_join"
    PATH_ARGS = ("room_id", "user_id")

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.federation_handler = hs.get_federation_handler()
@@ -320,7 +320,7 @@ class ReplicationUserJoinedLeftRoomRestServlet(ReplicationEndpoint):
    PATH_ARGS = ("room_id", "user_id", "change")
    CACHE = False  # No point caching as should return instantly.

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.registeration_handler = hs.get_registration_handler()
@@ -360,7 +360,7 @@ class ReplicationUserJoinedLeftRoomRestServlet(ReplicationEndpoint):
        return 200, {}


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationRemoteJoinRestServlet(hs).register(http_server)
    ReplicationRemoteRejectInviteRestServlet(hs).register(http_server)
    ReplicationUserJoinedLeftRoomRestServlet(hs).register(http_server)
@@ -117,6 +117,6 @@ class ReplicationPresenceSetState(ReplicationEndpoint):
        )


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationBumpPresenceActiveTime(hs).register(http_server)
    ReplicationPresenceSetState(hs).register(http_server)
@@ -67,5 +67,5 @@ class ReplicationRemovePusherRestServlet(ReplicationEndpoint):
        return 200, {}


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationRemovePusherRestServlet(hs).register(http_server)
@@ -13,10 +13,14 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.http.servlet import parse_json_object_from_request
 from synapse.replication.http._base import ReplicationEndpoint

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -26,7 +30,7 @@ class ReplicationRegisterServlet(ReplicationEndpoint):
    NAME = "register_user"
    PATH_ARGS = ("user_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)
        self.store = hs.get_datastore()
        self.registration_handler = hs.get_registration_handler()
@@ -100,7 +104,7 @@ class ReplicationPostRegisterActionsServlet(ReplicationEndpoint):
    NAME = "post_register"
    PATH_ARGS = ("user_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)
        self.store = hs.get_datastore()
        self.registration_handler = hs.get_registration_handler()
@@ -130,6 +134,6 @@ class ReplicationPostRegisterActionsServlet(ReplicationEndpoint):
        return 200, {}


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationRegisterServlet(hs).register(http_server)
    ReplicationPostRegisterActionsServlet(hs).register(http_server)
@@ -13,6 +13,7 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import make_event_from_dict
@@ -22,6 +23,9 @@ from synapse.replication.http._base import ReplicationEndpoint
 from synapse.types import Requester, UserID
 from synapse.util.metrics import Measure

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -57,7 +61,7 @@ class ReplicationSendEventRestServlet(ReplicationEndpoint):
    NAME = "send_event"
    PATH_ARGS = ("event_id",)

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self.event_creation_handler = hs.get_event_creation_handler()
@@ -135,5 +139,5 @@ class ReplicationSendEventRestServlet(ReplicationEndpoint):
        )


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationSendEventRestServlet(hs).register(http_server)
@@ -13,11 +13,15 @@
 # limitations under the License.

 import logging
+from typing import TYPE_CHECKING

 from synapse.api.errors import SynapseError
 from synapse.http.servlet import parse_integer
 from synapse.replication.http._base import ReplicationEndpoint

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -46,7 +50,7 @@ class ReplicationGetStreamUpdates(ReplicationEndpoint):
    PATH_ARGS = ("stream_name",)
    METHOD = "GET"

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__(hs)

        self._instance_name = hs.get_instance_name()
@@ -74,5 +78,5 @@ class ReplicationGetStreamUpdates(ReplicationEndpoint):
        )


-def register_servlets(hs, http_server):
+def register_servlets(hs: "HomeServer", http_server):
    ReplicationGetStreamUpdates(hs).register(http_server)
@@ -13,18 +13,21 @@
 # limitations under the License.

 import logging
-from typing import Optional
+from typing import TYPE_CHECKING, Optional

 from synapse.storage.database import DatabasePool
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
 from synapse.storage.engines import PostgresEngine
 from synapse.storage.util.id_generators import MultiWriterIdGenerator

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


 class BaseSlavedStore(CacheInvalidationWorkerStore):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)
        if isinstance(self.database_engine, PostgresEngine):
            self._cache_id_gen: Optional[
@@ -12,15 +12,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import TYPE_CHECKING
+
 from synapse.storage.database import DatabasePool
 from synapse.storage.databases.main.client_ips import LAST_SEEN_GRANULARITY
 from synapse.util.caches.lrucache import LruCache

 from ._base import BaseSlavedStore

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class SlavedClientIpStore(BaseSlavedStore):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)

        self.client_ip_last_seen: LruCache[tuple, int] = LruCache(
@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import TYPE_CHECKING
+
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
 from synapse.replication.tcp.streams._base import DeviceListsStream, UserSignatureStream
@@ -20,9 +22,12 @@ from synapse.storage.databases.main.devices import DeviceWorkerStore
 from synapse.storage.databases.main.end_to_end_keys import EndToEndKeyWorkerStore
 from synapse.util.caches.stream_change_cache import StreamChangeCache

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class SlavedDeviceStore(EndToEndKeyWorkerStore, DeviceWorkerStore, BaseSlavedStore):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)

        self.hs = hs
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
+from typing import TYPE_CHECKING

 from synapse.storage.database import DatabasePool
 from synapse.storage.databases.main.event_federation import EventFederationWorkerStore
@@ -30,6 +31,9 @@ from synapse.util.caches.stream_change_cache import StreamChangeCache

 from ._base import BaseSlavedStore

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -54,7 +58,7 @@ class SlavedEventStore(
    RelationsWorkerStore,
    BaseSlavedStore,
 ):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)

        events_max = self._stream_id_gen.get_current_token()
@@ -12,14 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import TYPE_CHECKING
+
 from synapse.storage.database import DatabasePool
 from synapse.storage.databases.main.filtering import FilteringStore

 from ._base import BaseSlavedStore

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class SlavedFilteringStore(BaseSlavedStore):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)

    # Filters are immutable so this cache doesn't need to be expired
@@ -12,6 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import TYPE_CHECKING
+
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
 from synapse.replication.tcp.streams import GroupServerStream
@@ -19,9 +21,12 @@ from synapse.storage.database import DatabasePool
 from synapse.storage.databases.main.group_server import GroupServerWorkerStore
 from synapse.util.caches.stream_change_cache import StreamChangeCache

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class SlavedGroupServerStore(GroupServerWorkerStore, BaseSlavedStore):
-    def __init__(self, database: DatabasePool, db_conn, hs):
+    def __init__(self, database: DatabasePool, db_conn, hs: "HomeServer"):
        super().__init__(database, db_conn, hs)

        self.hs = hs
@@ -21,6 +21,8 @@ from synapse.logging.context import make_deferred_yieldable
 from synapse.util import json_decoder, json_encoder

 if TYPE_CHECKING:
+    from txredisapi import RedisProtocol
+
    from synapse.server import HomeServer

 set_counter = Counter(
@@ -59,7 +61,12 @@ class ExternalCache:
    """

    def __init__(self, hs: "HomeServer"):
-        self._redis_connection = hs.get_outbound_redis_connection()
+        if hs.config.redis.redis_enabled:
+            self._redis_connection: Optional[
+                "RedisProtocol"
+            ] = hs.get_outbound_redis_connection()
+        else:
+            self._redis_connection = None

    def _get_redis_key(self, cache_name: str, key: str) -> str:
        return "cache_v1:%s:%s" % (cache_name, key)
@@ -294,7 +294,7 @@ class ReplicationCommandHandler:
            # This shouldn't be possible
            raise Exception("Unrecognised command %s in stream queue", cmd.NAME)

-    def start_replication(self, hs):
+    def start_replication(self, hs: "HomeServer"):
        """Helper method to start a replication connection to the remote server
        using TCP.
        """
@@ -321,6 +321,8 @@ class ReplicationCommandHandler:
                hs.config.redis.redis_host,  # type: ignore[arg-type]
                hs.config.redis.redis_port,
                self._factory,
+                timeout=30,
+                bindAddress=None,
            )
        else:
            client_name = hs.get_instance_name()
@@ -331,6 +333,8 @@ class ReplicationCommandHandler:
                host,  # type: ignore[arg-type]
                port,
                self._factory,
+                timeout=30,
+                bindAddress=None,
            )

    def get_streams(self) -> Dict[str, Stream]:
@@ -16,6 +16,7 @@

 import logging
 import random
+from typing import TYPE_CHECKING

 from prometheus_client import Counter

@@ -27,6 +28,9 @@ from synapse.replication.tcp.protocol import ServerReplicationStreamProtocol
 from synapse.replication.tcp.streams import EventsStream
 from synapse.util.metrics import Measure

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 stream_updates_counter = Counter(
    "synapse_replication_tcp_resource_stream_updates", "", ["stream_name"]
 )
@@ -37,7 +41,7 @@ logger = logging.getLogger(__name__)
 class ReplicationStreamProtocolFactory(Factory):
    """Factory for new replication connections."""

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.command_handler = hs.get_tcp_replication()
        self.clock = hs.get_clock()
        self.server_name = hs.config.server.server_name
@@ -65,7 +69,7 @@ class ReplicationStreamer:
    data is available it will propagate to all connected clients.
    """

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.store = hs.get_datastore()
        self.clock = hs.get_clock()
        self.notifier = hs.get_notifier()
@@ -241,7 +241,7 @@ class BackfillStream(Stream):
    NAME = "backfill"
    ROW_TYPE = BackfillStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -363,7 +363,7 @@ class ReceiptsStream(Stream):
    NAME = "receipts"
    ROW_TYPE = ReceiptsStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -380,7 +380,7 @@ class PushRulesStream(Stream):
    NAME = "push_rules"
    ROW_TYPE = PushRulesStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.store = hs.get_datastore()

        super().__init__(
@@ -405,7 +405,7 @@ class PushersStream(Stream):
    NAME = "pushers"
    ROW_TYPE = PushersStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()

        super().__init__(
@@ -438,7 +438,7 @@ class CachesStream(Stream):
    NAME = "caches"
    ROW_TYPE = CachesStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -459,7 +459,7 @@ class DeviceListsStream(Stream):
    NAME = "device_lists"
    ROW_TYPE = DeviceListsStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -476,7 +476,7 @@ class ToDeviceStream(Stream):
    NAME = "to_device"
    ROW_TYPE = ToDeviceStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -495,7 +495,7 @@ class TagAccountDataStream(Stream):
    NAME = "tag_account_data"
    ROW_TYPE = TagAccountDataStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -582,7 +582,7 @@ class GroupServerStream(Stream):
    NAME = "groups"
    ROW_TYPE = GroupsStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -599,7 +599,7 @@ class UserSignatureStream(Stream):
    NAME = "user_signature"
    ROW_TYPE = UserSignatureStreamRow

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        store = hs.get_datastore()
        super().__init__(
            hs.get_instance_name(),
@@ -110,7 +110,7 @@ class DevicesRestServlet(RestServlet):
    def __init__(self, hs: "HomeServer"):
        """
        Args:
-            hs (synapse.server.HomeServer): server
+            hs: server
        """
        self.hs = hs
        self.auth = hs.get_auth()
@@ -326,6 +326,9 @@ class UserRestServletV2(RestServlet):
                        target_user.to_string()
                    )

+            if "user_type" in body:
+                await self.store.set_user_type(target_user, user_type)
+
            user = await self.admin_handler.get_user(target_user)
            assert user is not None

@@ -128,9 +128,10 @@ class RelationSendServlet(RestServlet):

        content["m.relates_to"] = {
            "event_id": parent_id,
-            "key": aggregation_key,
            "rel_type": relation_type,
        }
+        if aggregation_key is not None:
+            content["m.relates_to"]["key"] = aggregation_key

        event_dict = {
            "type": event_type,
@@ -718,9 +718,12 @@ def decode_body(
    if not body:
        return None

+    # The idea here is that multiple encodings are tried until one works.
+    # Unfortunately the result is never used and then LXML will decode the string
+    # again with the found encoding.
    for encoding in get_html_media_encodings(body, content_type):
        try:
-            body_str = body.decode(encoding)
+            body.decode(encoding)
        except Exception:
            pass
        else:
@@ -732,11 +735,11 @@ def decode_body(
    from lxml import etree

    # Create an HTML parser.
-    parser = etree.HTMLParser(recover=True, encoding="utf-8")
+    parser = etree.HTMLParser(recover=True, encoding=encoding)

    # Attempt to parse the body. Returns None if the body was successfully
    # parsed, but no tree was found.
-    return etree.fromstring(body_str, parser)
+    return etree.fromstring(body, parser)


 def _calc_og(tree: "etree.Element", media_uri: str) -> Dict[str, Optional[str]]:
@@ -800,9 +800,14 @@ class HomeServer(metaclass=abc.ABCMeta):
        return ExternalCache(self)

    @cache_in_self
-    def get_outbound_redis_connection(self) -> Optional["RedisProtocol"]:
-        if not self.config.redis.redis_enabled:
-            return None
+    def get_outbound_redis_connection(self) -> "RedisProtocol":
+        """
+        The Redis connection used for replication.
+
+        Raises:
+            AssertionError: if Redis is not enabled in the homeserver config.
+        """
+        assert self.config.redis.redis_enabled

        # We only want to import redis module if we're using it, as we have
        # `txredisapi` as an optional dependency.
@@ -101,6 +101,7 @@ class BackgroundUpdater:
        self._all_done = False

    def start_doing_background_updates(self) -> None:
+        return
        run_as_background_process("background_updates", self.run_background_updates)

    async def run_background_updates(self, sleep: bool = True) -> None:
@@ -19,6 +19,7 @@ from collections import defaultdict
 from sys import intern
 from time import monotonic as monotonic_time
 from typing import (
+    TYPE_CHECKING,
    Any,
    Callable,
    Collection,
@@ -52,6 +53,9 @@ from synapse.storage.background_updates import BackgroundUpdater
 from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine, Sqlite3Engine
 from synapse.storage.types import Connection, Cursor

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 # python 3 does not have a maximum int value
 MAX_TXN_ID = 2 ** 63 - 1

@@ -392,7 +396,7 @@ class DatabasePool:

    def __init__(
        self,
-        hs,
+        hs: "HomeServer",
        database_config: DatabaseConnectionConfig,
        engine: BaseDatabaseEngine,
    ):
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Richard van der Hoff	1894419915	disable background updates	2021-11-04 16:08:48 +00:00
Erik Johnston	2d44ee6868	Update changelog	2021-11-02 13:25:42 +00:00
Erik Johnston	df84ad602b	1.46.0	2021-11-02 13:23:01 +00:00
Richard van der Hoff	576921c66a	Force deb compression with `xz`. (#11197 ) Fixes a problem where `impish` packages could not be processed by `reprepro`.	2021-10-27 17:06:32 +01:00
Patrick Cloke	b3e843be88	Fix URL preview errors when previewing XML documents. (#11196 )	2021-10-27 14:48:02 +00:00
Erik Johnston	e0ef8fe58d	Update release date	2021-10-27 14:44:45 +01:00
Erik Johnston	b615fc35d6	Update changelog with new changes	2021-10-27 13:22:48 +01:00
Sean Quah	f3a4be8700	Shut down the DNS threadpool (#11190 ) The DNS threadpool must be explicitly stopped, otherwise Synapse will hang indefinitely when asked to shut down.	2021-10-27 13:04:56 +01:00
Erik Johnston	72626b78ef	Fix thread BG update to not seq scan event_json (#11192 ) For some reason the query optimiser decided to seq scan both tables, rather than index scanning `event_json`.	2021-10-27 11:33:21 +00:00
Sean Quah	2dbef6c10a	Update CHANGES.md	2021-10-26 17:54:25 +01:00
Sean Quah	60ad9460c4	Move #10975 to bugfix section in changelog	2021-10-26 14:26:36 +01:00
Sean Quah	400f391f71	Update CHANGES.md	2021-10-26 14:21:09 +01:00
Sean Quah	34b0222c2b	Update CHANGES.md	2021-10-26 14:08:52 +01:00
Sean Quah	cc75a6b1b2	1.46.0rc1	2021-10-26 14:04:51 +01:00
Erik Johnston	7004f43da1	Move DNS lookups into separate thread pool (#11177 ) This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.	2021-10-26 13:45:38 +01:00
Patrick Cloke	d52c58dfa3	Add a background update for updating MSC3440 relation threads. (#11181 )	2021-10-26 07:38:45 -04:00
Brendan Abolivier	8c8e36af0d	Document the version each module API method was added to Synapse (#11183 )	2021-10-26 11:09:10 +02:00
Jason Robinson	63cbdd8af0	Enable changing user type via users admin API (#11174 ) Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2021-10-26 09:01:06 +00:00
Brendan Abolivier	c1510c97b5	Fix cyclic import in the module API (#11180 ) Introduced in #10548 See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.	2021-10-25 16:45:19 +00:00
AndrewFerr	4387b791e0	Don't set new room alias before potential 403 (#10930 ) Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>	2021-10-25 15:24:49 +01:00
Richard van der Hoff	da957a60e8	Ensure that we correctly auth events returned by `send_join` (#11012 ) This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected. To do this I've reused the existing code we use when persisting outliers elsewhere. I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.	2021-10-25 15:21:09 +01:00
Sean Quah	85a09f8b8b	Fix module API's `get_user_ip_and_agents` function when run on workers (#11112 )	2021-10-25 13:01:04 +01:00
Sean Quah	2b82ec425f	Add type hints for most `HomeServer` parameters (#11095 )	2021-10-22 18:15:41 +01:00
Jason Robinson	b9ce53e878	Fix synapse.config module "read" command (#11145 ) `synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse. The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example. Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2021-10-22 12:00:52 +02:00
Dirk Klimpel	b0f03aeb6a	Add more information what happens when a user is deactivated (#11083 )	2021-10-22 09:00:28 +01:00
Patrick Cloke	ba00e20234	Add a thread relation type per MSC3440. (#11088 ) Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).	2021-10-21 14:39:16 -04:00
				`@@ -1 +0,0 @@`
				`Port the Password Auth Providers module interface to the new generic interface.`
				`@@ -1 +0,0 @@`
				Add type hints to `synapse.storage.databases.main.client_ips`.
				`@@ -1 +0,0 @@`
				Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch.
				`@@ -1 +0,0 @@`
				Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection.
				`@@ -1 +0,0 @@`
				`Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.`
				`@@ -1 +0,0 @@`
				`Include rejected status when we log events.`
				`@@ -1 +0,0 @@`
				`Add some extra logging to the event persistence code.`
				`@@ -1 +0,0 @@`
				`Add support for Ubuntu 21.10 "Impish Indri".`
				`@@ -1 +0,0 @@`
				Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room.