WIP docs

Allow /typing to be handled by any worker
Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407 )
2021-07-28 11:06:24 +01:00 · 2021-07-28 10:58:45 +01:00 · 2021-07-27 14:32:05 +01:00 · 2021-07-27 12:08:51 +00:00 · 2021-07-27 13:45:10 +02:00 · 2021-07-27 05:28:20 +00:00
114 changed files with 1895 additions and 969 deletions
@@ -12,6 +12,10 @@ on:
    # we do the full build on tags.
    tags: ["v*"]

+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+  
 permissions:
  contents: write

@@ -44,12 +48,43 @@ jobs:
        distro: ${{ fromJson(needs.get-distros.outputs.distros) }}

    steps:
-      - uses: actions/checkout@v2
+      - name: Checkout
+        uses: actions/checkout@v2
        with:
          path: src
-      - uses: actions/setup-python@v2
-      - run: ./src/scripts-dev/build_debian_packages "${{ matrix.distro }}"
-      - uses: actions/upload-artifact@v2
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v1
+        with:
+          install: true
+
+      - name: Set up docker layer caching
+        uses: actions/cache@v2
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+
+      - name: Set up python
+        uses: actions/setup-python@v2
+
+      - name: Build the packages
+        # see https://github.com/docker/build-push-action/issues/252
+        # for the cache magic here
+        run: |
+          ./src/scripts-dev/build_debian_packages \
+            --docker-build-arg=--cache-from=type=local,src=/tmp/.buildx-cache \
+            --docker-build-arg=--cache-to=type=local,mode=max,dest=/tmp/.buildx-cache-new \
+            --docker-build-arg=--progress=plain \
+            --docker-build-arg=--load \
+            "${{ matrix.distro }}"
+          rm -rf /tmp/.buildx-cache
+          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
+
+      - name: Upload debs as artifacts
+        uses: actions/upload-artifact@v2
        with:
          name: debs
          path: debs/*
@@ -5,6 +5,10 @@ on:
    branches: ["develop", "release-*"]
  pull_request:

+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+  
 jobs:
  lint:
    runs-on: ubuntu-latest
@@ -340,14 +344,19 @@ jobs:
        working-directory: complement/dockerfiles

      # Run Complement
-      - run: go test -v -tags synapse_blacklist,msc2403,msc2946,msc3083 ./tests
+      - run: go test -v -tags synapse_blacklist,msc2403,msc2946,msc3083 ./tests/...
        env:
          COMPLEMENT_BASE_IMAGE: complement-synapse:latest
        working-directory: complement

  # a job which marks all the other jobs as complete, thus allowing PRs to be merged.
  tests-done:
+    if: ${{ always() }}
    needs:
+      - lint
+      - lint-crlf
+      - lint-newsfile
+      - lint-sdist
      - trial
      - trial-olddeps
      - sytest
@@ -355,4 +364,16 @@ jobs:
      - complement
    runs-on: ubuntu-latest
    steps:
-      - run: "true"    
+      - name: Set build result
+        env:
+          NEEDS_CONTEXT: ${{ toJSON(needs) }}
+        # the `jq` incantation dumps out a series of "<job> <result>" lines
+        run: |
+          set -o pipefail
+          jq -r 'to_entries[] | [.key,.value.result] | join(" ")' \
+                          <<< $NEEDS_CONTEXT |
+              while read job result; do
+                  if [ "$result" != "success" ]; then
+                      echo "::set-failed ::Job $job returned $result"
+                  fi
+              done
@@ -1,31 +1,10 @@
-Synapse 1.39.0 (2021-07-29)
-===========================
-
-No significant changes.
-
-
-Synapse 1.39.0rc3 (2021-07-28)
+Synapse 1.39.0rc2 (2021-07-22)
 ==============================

 Bugfixes
 --------

- Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477))
- Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485))
- Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486))
-
-
-Internal Changes
----------------
-
- Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461))
- Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
-
-
-Synapse 1.39.0rc2 (2021-07-22)
-==============================
-
-This release also includes the changes in v1.38.1.
+- Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))


 Internal Changes
@@ -392,7 +392,7 @@ By now, you know the drill!
 # Notes for maintainers on merging PRs etc

 There are some notes for those with commit access to the project on how we
-manage git [here](docs/dev/git.md).
+manage git [here](docs/development/git.md).

 # Conclusion

@@ -34,14 +34,6 @@ additional-css = [
    "docs/website_files/table-of-contents.css",
    "docs/website_files/remove-nav-buttons.css",
    "docs/website_files/indent-section-headers.css",
-    "docs/website_files/version-picker.css",
 ]
-additional-js = [
-    "docs/website_files/table-of-contents.js",
-    "docs/website_files/version-picker.js",
-    "docs/website_files/version.js",
-]
-theme = "docs/website_files/theme"
-
-[preprocessor.schema_versions]
-command = "./scripts-dev/schema_versions.py"
+additional-js = ["docs/website_files/table-of-contents.js"]
+theme = "docs/website_files/theme"
@@ -0,0 +1 @@
+Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events.
@@ -0,0 +1 @@
+Initial support for MSC3244, Room version capabilities over the /capabilities API.
@@ -0,0 +1 @@
+Add a buffered logging handler which periodically flushes itself.
@@ -0,0 +1 @@
+Add type hints to `synapse.federation.transport.client` module.
@@ -0,0 +1 @@
+Email notifications now state whether an invitation is to a room or a space.
@@ -0,0 +1 @@
+Drop backwards-compatibility code that was required to support Ubuntu Xenial.
@@ -0,0 +1 @@
+Use a docker image cache for the prerequisites for the debian package build.
@@ -0,0 +1 @@
+Connect historical chunks together with chunk events instead of a content field (MSC2716).
@@ -0,0 +1 @@
+Improve servlet type hints.
@@ -0,0 +1 @@
+Improve servlet type hints.
@@ -0,0 +1 @@
+Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages.
@@ -0,0 +1 @@
+Update the `tests-done` Github Actions status.
@@ -0,0 +1 @@
+Fix hierarchy of providers on the OpenID page.
@@ -0,0 +1 @@
+Update type annotations to work with forthcoming Twisted 21.7.0 release.
@@ -0,0 +1 @@
+Add `creation_ts` to list users admin API.
@@ -0,0 +1 @@
+Cancel redundant GHA workflows when a new commit is pushed.
@@ -0,0 +1 @@
+Consolidate development documentation to `docs/development/`.
@@ -0,0 +1 @@
+Fix an error which prevented the Github Actions workflow to build the docker images from running.
@@ -0,0 +1 @@
+Disable `msc2716` Complement tests until Complement updates are merged.
@@ -0,0 +1 @@
+Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header.
@@ -0,0 +1 @@
+Additional type hints in the state handler.
@@ -0,0 +1 @@
+Update syntax used to run complement tests.
@@ -0,0 +1 @@
+Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`.
@@ -33,13 +33,11 @@ esac
 # Use --builtin-venv to use the better `venv` module from CPython 3.4+ rather
 # than the 2/3 compatible `virtualenv`.

-# Pin pip to 20.3.4 to fix breakage in 21.0 on py3.5 (xenial)
-
 dh_virtualenv \
    --install-suffix "matrix-synapse" \
    --builtin-venv \
    --python "$SNAKE" \
-    --upgrade-pip-to="20.3.4" \
+    --upgrade-pip \
    --preinstall="lxml" \
    --preinstall="mock" \
    --extra-pip-arg="--no-cache-dir" \
@@ -1,14 +1,8 @@
-matrix-synapse-py3 (1.39.0) stable; urgency=medium
+matrix-synapse-py3 (1.39.0ubuntu1) UNRELEASED; urgency=medium

-  * New synapse release 1.39.0.
+  * Drop backwards-compatibility code that was required to support Ubuntu Xenial.

- -- Synapse Packaging team <packages@matrix.org>  Thu, 29 Jul 2021 09:59:00 +0100
-
-matrix-synapse-py3 (1.39.0~rc3) stable; urgency=medium
-
-  * New synapse release 1.39.0~rc3.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 28 Jul 2021 13:30:58 +0100
+ -- Richard van der Hoff <richard@matrix.org>  Tue, 20 Jul 2021 00:10:03 +0100

 matrix-synapse-py3 (1.38.1) stable; urgency=medium

@@ -1 +1 @@
-9
+10
@@ -3,11 +3,8 @@ Section: contrib/python
 Priority: extra
 Maintainer: Synapse Packaging team <packages@matrix.org>
 # keep this list in sync with the build dependencies in docker/Dockerfile-dhvirtualenv.
-# TODO: Remove the dependency on dh-systemd after dropping support for Ubuntu xenial
-#       On all other supported releases, it's merely a transitional package which
-#       does nothing but depends on debhelper (> 9.20160709)
 Build-Depends:
- debhelper (>= 9.20160709) | dh-systemd,
+ debhelper (>= 10),
 dh-virtualenv (>= 1.1),
 libsystemd-dev,
 libpq-dev,
@@ -51,7 +51,5 @@ override_dh_shlibdeps:
 override_dh_virtualenv:
 	./debian/build_virtualenv

-# We are restricted to compat level 9 (because xenial), so have to
-# enable the systemd bits manually.
 %:
-	dh $@ --with python-virtualenv --with systemd
+	dh $@ --with python-virtualenv
@@ -15,6 +15,15 @@ ARG distro=""
 ###
 ### Stage 0: build a dh-virtualenv
 ###
+
+# This is only really needed on bionic and focal, since other distributions we
+# care about have a recent version of dh-virtualenv by default. Unfortunately,
+# it looks like focal is going to be with us for a while.
+#
+# (focal doesn't have a dh-virtualenv package at all. There is a PPA at
+# https://launchpad.net/~jyrki-pulliainen/+archive/ubuntu/dh-virtualenv, but
+# it's not obviously easier to use that than to build our own.)
+
 FROM ${distro} as builder

 RUN apt-get update -qq -o Acquire::Languages=none
@@ -27,7 +36,7 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \
        wget

 # fetch and unpack the package
-# TODO: Upgrade to 1.2.2 once xenial is dropped
+# TODO: Upgrade to 1.2.2 once bionic is dropped (1.2.2 requires debhelper 12; bionic has only 11)
 RUN mkdir /dh-virtualenv
 RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/spotify/dh-virtualenv/archive/ac6e1b1.tar.gz
 RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz
@@ -59,8 +68,6 @@ ENV LANG C.UTF-8
 #
 # NB: keep this list in sync with the list of build-deps in debian/control
 # TODO: it would be nice to do that automatically.
-# TODO: Remove the dh-systemd stanza after dropping support for Ubuntu xenial
-#       it's a transitional package on all other, more recent releases
 RUN apt-get update -qq -o Acquire::Languages=none \
    && env DEBIAN_FRONTEND=noninteractive apt-get install \
        -yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
@@ -76,10 +83,7 @@ RUN apt-get update -qq -o Acquire::Languages=none \
        python3-venv \
        sqlite3 \
        libpq-dev \
-        xmlsec1 \
-    && ( env DEBIAN_FRONTEND=noninteractive apt-get install \
-         -yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
-         dh-systemd || true )
+        xmlsec1

 COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /

@@ -67,7 +67,7 @@
 # Development
  - [Contributing Guide](development/contributing_guide.md)
  - [Code Style](code_style.md)
-  - [Git Usage](dev/git.md)
+  - [Git Usage](development/git.md)
  - [Testing]()
  - [OpenTracing](opentracing.md)
  - [Database Schemas](development/database_schema.md)
@@ -77,8 +77,8 @@
    - [TCP Replication](tcp_replication.md)
  - [Internal Documentation](development/internal_documentation/README.md)
    - [Single Sign-On]()
-      - [SAML](dev/saml.md)
-      - [CAS](dev/cas.md)
+      - [SAML](development/saml.md)
+      - [CAS](development/cas.md)
    - [State Resolution]()
      - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
    - [Media Repository](media_repository.md)
@@ -144,7 +144,8 @@ A response body like the following is returned:
            "deactivated": 0,
            "shadow_banned": 0,
            "displayname": "<User One>",
-            "avatar_url": null
+            "avatar_url": null,
+            "creation_ts": 1560432668000
        }, {
            "name": "<user_id2>",
            "is_guest": 0,
@@ -153,7 +154,8 @@ A response body like the following is returned:
            "deactivated": 0,
            "shadow_banned": 0,
            "displayname": "<User Two>",
-            "avatar_url": "<avatar_url>"
+            "avatar_url": "<avatar_url>",
+            "creation_ts": 1561550621000
        }
    ],
    "next_token": "100",
@@ -197,11 +199,12 @@ The following parameters should be set in the URL:
  - `shadow_banned` - Users are ordered by `shadow_banned` status.
  - `displayname` - Users are ordered alphabetically by `displayname`.
  - `avatar_url` - Users are ordered alphabetically by avatar URL.
+  - `creation_ts` - Users are ordered by when the users was created in ms.

 - `dir` - Direction of media order. Either `f` for forwards or `b` for backwards.
  Setting this value to `b` will reverse the above sort order. Defaults to `f`.

-Caution. The database only has indexes on the columns `name` and `created_ts`.
+Caution. The database only has indexes on the columns `name` and `creation_ts`.
 This means that if a different sort order is used (`is_guest`, `admin`,
 `user_type`, `deactivated`, `shadow_banned`, `avatar_url` or `displayname`),
 this can cause a large load on the database, especially for large environments.
@@ -222,6 +225,7 @@ The following fields are returned in the JSON response body:
  - `shadow_banned` - bool - Status if that user has been marked as shadow banned.
  - `displayname` - string - The user's display name if they have set one.
  - `avatar_url` - string -  The user's avatar URL if they have set one.
+  - `creation_ts` - integer - The user's creation timestamp in ms.

 - `next_token`: string representing a positive integer - Indication for pagination. See above.
 - `total` - integer - Total number of media.
@@ -9,7 +9,7 @@ commits each of which contains a single change building on what came
 before. Here, by way of an arbitrary example, is the top of `git log --graph
 b2dba0607`:

-<img src="git/clean.png" alt="clean git graph" width="500px">
+<img src="img/git/clean.png" alt="clean git graph" width="500px">

 Note how the commit comment explains clearly what is changing and why. Also
 note the *absence* of merge commits, as well as the absence of commits called
@@ -61,7 +61,7 @@ Ok, so that's what we'd like to achieve. How do we achieve it?
 The TL;DR is: when you come to merge a pull request, you *probably* want to
 “squash and merge”:

-![squash and merge](git/squash.png).
+![squash and merge](img/git/squash.png).

 (This applies whether you are merging your own PR, or that of another
 contributor.)
@@ -105,7 +105,7 @@ complicated. Here's how we do it.

 Let's start with a picture:

-![branching model](git/branches.jpg)
+![branching model](img/git/branches.jpg)

 It looks complicated, but it's really not. There's one basic rule: *anyone* is
 free to merge from *any* more-stable branch to *any* less-stable branch at
@@ -410,7 +410,7 @@ oidc_providers:
        display_name_template: "{{ user.name }}"
 ```

-## Apple
+### Apple

 Configuring "Sign in with Apple" (SiWA) requires an Apple Developer account.

@@ -28,7 +28,7 @@ handlers:
    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
    # logs will still be flushed immediately.
    buffer:
-        class: logging.handlers.MemoryHandler
+        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
        target: file
        # The capacity is the number of log lines that are buffered before
        # being written to disk. Increasing this will lead to better
@@ -36,6 +36,9 @@ handlers:
        # be written to disk.
        capacity: 10
        flushLevel: 30  # Flush for WARNING logs as well
+        # The period of time, in seconds, between forced flushes.
+        # Messages will not be delayed for longer than this time.
+        period: 5

    # A handler that writes logs to stderr. Unused by default, but can be used
    # instead of "buffer" and "file" in the logger handlers.
@@ -24,11 +24,6 @@ Finally, we also stylise the chapter titles in the left sidebar by indenting the
 slightly so that they are more visually distinguishable from the section headers
 (the bold titles). This is done through the `indent-section-headers.css` file.

-In addition to these modifications, we have added a version picker to the documentation.
-Users can switch between documentations for different versions of Synapse.
-This functionality was implemented through the `version-picker.js` and
-`version-picker.css` files.
-
 More information can be found in mdbook's official documentation for
 [injecting page JS/CSS](https://rust-lang.github.io/mdBook/format/config.html)
 and
@@ -131,18 +131,6 @@
                            <i class="fa fa-search"></i>
                        </button>
                        {{/if}}
-                        <div class="version-picker">
-                            <div class="dropdown">
-                                <div class="select">
-                                    <span></span>
-                                    <i class="fa fa-chevron-down"></i>
-                                </div>
-                                <input type="hidden" name="version">
-                                <ul class="dropdown-menu">
-                                    <!-- Versions will be added dynamically in version-picker.js -->
-                                </ul>
-                            </div>
-                        </div>      
                    </div>

                    <h1 class="menu-title">{{ book_title }}</h1>
@@ -321,4 +309,4 @@
        {{/if}}

    </body>
-</html>
+</html>
@@ -1,78 +0,0 @@
-.version-picker {
-    display: flex;
-    align-items: center;
-}
-
-.version-picker .dropdown {
-    width: 130px;
-    max-height: 29px;
-    margin-left: 10px;
-    display: inline-block;
-    border-radius: 4px;
-    border: 1px solid var(--theme-popup-border);
-    position: relative;
-    font-size: 13px;
-    color: var(--fg);
-    height: 100%;
-    text-align: left;
-}
-.version-picker .dropdown .select {
-    cursor: pointer;
-    display: block;
-    padding: 5px 2px 5px 15px;
-}
-.version-picker .dropdown .select > i {
-    font-size: 10px;
-    color: var(--fg);
-    cursor: pointer;
-    float: right;
-    line-height: 20px !important;
-}
-.version-picker .dropdown:hover {
-    border: 1px solid var(--theme-popup-border);
-}
-.version-picker .dropdown:active {
-    background-color: var(--theme-popup-bg);
-}
-.version-picker .dropdown.active:hover,
-.version-picker .dropdown.active {
-    border: 1px solid var(--theme-popup-border);
-    border-radius: 2px 2px 0 0;
-    background-color: var(--theme-popup-bg);
-}
-.version-picker .dropdown.active .select > i {
-    transform: rotate(-180deg);
-}
-.version-picker .dropdown .dropdown-menu {
-    position: absolute;
-    background-color: var(--theme-popup-bg);
-    width: 100%;
-    left: -1px;
-    right: 1px;
-    margin-top: 1px;
-    border: 1px solid var(--theme-popup-border);
-    border-radius: 0 0 4px 4px;
-    overflow: hidden;
-    display: none;
-    max-height: 300px;
-    overflow-y: auto;
-    z-index: 9;
-}
-.version-picker .dropdown .dropdown-menu li {
-    font-size: 12px;
-    padding: 6px 20px;
-    cursor: pointer;
-} 
-.version-picker .dropdown .dropdown-menu {
-    padding: 0;
-    list-style: none;
-}
-.version-picker .dropdown .dropdown-menu li:hover {
-    background-color: var(--theme-hover);
-}
-.version-picker .dropdown .dropdown-menu li.active::before {
-    display: inline-block;
-    content: "✓";
-    margin-inline-start: -14px;
-    width: 14px;
-}
@@ -1,127 +0,0 @@
-
-const dropdown = document.querySelector('.version-picker .dropdown');
-const dropdownMenu = dropdown.querySelector('.dropdown-menu');
-
-fetchVersions(dropdown, dropdownMenu).then(() => {
-    initializeVersionDropdown(dropdown, dropdownMenu);
-});
-
-/**
- * Initialize the dropdown functionality for version selection.
- * 
- * @param {Element} dropdown - The dropdown element.
- * @param {Element} dropdownMenu - The dropdown menu element.
- */
-function initializeVersionDropdown(dropdown, dropdownMenu) {
-    // Toggle the dropdown menu on click
-    dropdown.addEventListener('click', function () {
-        this.setAttribute('tabindex', 1);
-        this.classList.toggle('active');
-        dropdownMenu.style.display = (dropdownMenu.style.display === 'block') ? 'none' : 'block';
-    });
-  
-    // Remove the 'active' class and hide the dropdown menu on focusout
-    dropdown.addEventListener('focusout', function () {
-        this.classList.remove('active');
-        dropdownMenu.style.display = 'none';
-    });
-  
-    // Handle item selection within the dropdown menu
-    const dropdownMenuItems = dropdownMenu.querySelectorAll('li');    
-    dropdownMenuItems.forEach(function (item) {
-        item.addEventListener('click', function () {
-            dropdownMenuItems.forEach(function (item) {
-                item.classList.remove('active');
-            });
-            this.classList.add('active');
-            dropdown.querySelector('span').textContent = this.textContent;
-            dropdown.querySelector('input').value = this.getAttribute('id');
-
-            window.location.href = changeVersion(window.location.href, this.textContent);
-        });
-    });
-};
-
-/**
- * This function fetches the available versions from a GitHub repository
- * and inserts them into the version picker.
- * 
- * @param {Element} dropdown - The dropdown element.
- * @param {Element} dropdownMenu - The dropdown menu element.
- * @returns {Promise<Array<string>>} A promise that resolves with an array of available versions.
- */
-function fetchVersions(dropdown, dropdownMenu) {
-    return new Promise((resolve, reject) => {
-        window.addEventListener("load", () => {
-
-            fetch("https://api.github.com/repos/matrix-org/synapse/git/trees/gh-pages", {
-                cache: "force-cache",
-            }).then(res => 
-                res.json()
-            ).then(resObject => {
-                const excluded = ['dev-docs', 'v1.91.0', 'v1.80.0', 'v1.69.0'];
-                const tree = resObject.tree.filter(item => item.type === "tree" && !excluded.includes(item.path));
-                const versions = tree.map(item => item.path).sort(sortVersions);
-
-                // Create a list of <li> items for versions
-                versions.forEach((version) => {
-                    const li = document.createElement("li");
-                    li.textContent = version;
-                    li.id = version;
-    
-                    if (window.SYNAPSE_VERSION === version) {
-                        li.classList.add('active');
-                        dropdown.querySelector('span').textContent = version;
-                        dropdown.querySelector('input').value = version;
-                    }
-    
-                    dropdownMenu.appendChild(li);
-                });
-
-                resolve(versions);
-
-            }).catch(ex => {
-                console.error("Failed to fetch version data", ex);
-                reject(ex);
-            })
-        });
-    });
-}
-
-/**
- * Custom sorting function to sort an array of version strings.
- *
- * @param {string} a - The first version string to compare.
- * @param {string} b - The second version string to compare.
- * @returns {number} - A negative number if a should come before b, a positive number if b should come before a, or 0 if they are equal.
- */
-function sortVersions(a, b) {
-    // Put 'develop' and 'latest' at the top
-    if (a === 'develop' || a === 'latest') return -1;
-    if (b === 'develop' || b === 'latest') return 1;
-
-    const versionA = (a.match(/v\d+(\.\d+)+/) || [])[0];
-    const versionB = (b.match(/v\d+(\.\d+)+/) || [])[0];
-
-    return versionB.localeCompare(versionA);
-}
-
-/**
- * Change the version in a URL path.
- *
- * @param {string} url - The original URL to be modified.
- * @param {string} newVersion - The new version to replace the existing version in the URL.
- * @returns {string} The updated URL with the new version.
- */
-function changeVersion(url, newVersion) {
-    const parsedURL = new URL(url);
-    const pathSegments = parsedURL.pathname.split('/');
-  
-    // Modify the version
-    pathSegments[2] = newVersion;
-
-    // Reconstruct the URL
-    parsedURL.pathname = pathSegments.join('/');
-  
-    return parsedURL.href;
-}
@@ -1 +0,0 @@
-window.SYNAPSE_VERSION = 'v1.39';
@@ -319,11 +319,24 @@ effects of bursts of events from that bridge on events sent by normal users.

 #### Stream writers

-Additionally, there is *experimental* support for moving writing of specific
-streams (such as events) off of the main process to a particular worker. (This
-is only supported with Redis-based replication.)
+Additionally, there is support for moving writing of specific streams (such as
+events) off of the main process to a particular worker. (This is only supported
+with Redis-based replication.)

-Currently supported streams are `events` and `typing`.
+Currently supported streams are, and which endpoints **must** be routed to them:
+  * `events`
+
+  * `typing`:
+    * `^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/typing`
+
+  * `to_device`:
+    `^/_matrix/client/(api/v1|r0|unstable)/sendToDevice/`
+    `^/_matrix/client/(api/v1|r0|unstable)/keys/claim`
+    `^/_matrix/client/(api/v1|r0|unstable)/room_keys`
+
+  * `account_data`
+  * `receipts`
+  * `presence`

 To enable this, the worker must have a HTTP replication listener configured,
 have a `worker_name` and be listed in the `instance_map` config. For example to
@@ -340,10 +353,10 @@ stream_writers:
    events: event_persister1
 ```

-The `events` stream also experimentally supports having multiple writers, where
-work is sharded between them by room ID. Note that you *must* restart all worker
-instances when adding or removing event persisters. An example `stream_writers`
-configuration with multiple writers:
+The `events` stream also supports having multiple writers, where work is sharded
+between them by room ID. Note that you *must* restart all worker instances when
+adding or removing event persisters. An example `stream_writers` configuration
+with multiple writers:

 ```yaml
 stream_writers:
@@ -352,6 +365,8 @@ stream_writers:
        - event_persister2
 ```

+All other streams currently only support having a single writer.
+
 #### Background tasks

 There is also *experimental* support for moving background tasks to a separate
@@ -17,6 +17,7 @@ import subprocess
 import sys
 import threading
 from concurrent.futures import ThreadPoolExecutor
+from typing import Optional, Sequence

 DISTS = (
    "debian:buster",
@@ -39,8 +40,11 @@ projdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))


 class Builder(object):
-    def __init__(self, redirect_stdout=False):
+    def __init__(
+        self, redirect_stdout=False, docker_build_args: Optional[Sequence[str]] = None
+    ):
        self.redirect_stdout = redirect_stdout
+        self._docker_build_args = tuple(docker_build_args or ())
        self.active_containers = set()
        self._lock = threading.Lock()
        self._failed = False
@@ -79,8 +83,8 @@ class Builder(object):
            stdout = None

        # first build a docker image for the build environment
-        subprocess.check_call(
-            [
+        build_args = (
+            (
                "docker",
                "build",
                "--tag",
@@ -89,8 +93,13 @@ class Builder(object):
                "distro=" + dist,
                "-f",
                "docker/Dockerfile-dhvirtualenv",
-                "docker",
-            ],
+            )
+            + self._docker_build_args
+            + ("docker",)
+        )
+
+        subprocess.check_call(
+            build_args,
            stdout=stdout,
            stderr=subprocess.STDOUT,
            cwd=projdir,
@@ -147,9 +156,7 @@ class Builder(object):
                self.active_containers.remove(c)


-def run_builds(dists, jobs=1, skip_tests=False):
-    builder = Builder(redirect_stdout=(jobs > 1))
-
+def run_builds(builder, dists, jobs=1, skip_tests=False):
    def sig(signum, _frame):
        print("Caught SIGINT")
        builder.kill_containers()
@@ -180,6 +187,11 @@ if __name__ == "__main__":
        action="store_true",
        help="skip running tests after building",
    )
+    parser.add_argument(
+        "--docker-build-arg",
+        action="append",
+        help="specify an argument to pass to docker build",
+    )
    parser.add_argument(
        "--show-dists-json",
        action="store_true",
@@ -195,4 +207,12 @@ if __name__ == "__main__":
    if args.show_dists_json:
        print(json.dumps(DISTS))
    else:
-        run_builds(dists=args.dist, jobs=args.jobs, skip_tests=args.no_check)
+        builder = Builder(
+            redirect_stdout=(args.jobs > 1), docker_build_args=args.docker_build_arg
+        )
+        run_builds(
+            builder,
+            dists=args.dist,
+            jobs=args.jobs,
+            skip_tests=args.no_check,
+        )
@@ -65,4 +65,4 @@ if [[ -n "$1" ]]; then
 fi

 # Run the tests!
-go test -v -tags synapse_blacklist,msc2946,msc3083,msc2716,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests
+go test -v -tags synapse_blacklist,msc2946,msc3083,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests/...
@@ -139,11 +139,6 @@ def run():

    # Switch to the release branch.
    parsed_new_version = version.parse(new_version)
-
-    # We assume for debian changelogs that we only do RCs or full releases.
-    assert not parsed_new_version.is_devrelease
-    assert not parsed_new_version.is_postrelease
-
    release_branch_name = (
        f"release-v{parsed_new_version.major}.{parsed_new_version.minor}"
    )
@@ -195,21 +190,12 @@ def run():
    # Generate changelogs
    subprocess.run("python3 -m towncrier", shell=True)

-    # Generate debian changelogs
-    if parsed_new_version.pre is not None:
-        # If this is an RC then we need to coerce the version string to match
-        # Debian norms, e.g. 1.39.0rc2 gets converted to 1.39.0~rc2.
-        base_ver = parsed_new_version.base_version
-        pre_type, pre_num = parsed_new_version.pre
-        debian_version = f"{base_ver}~{pre_type}{pre_num}"
-    else:
-        debian_version = new_version
-
-    subprocess.run(
-        f'dch -M -v {debian_version} "New synapse release {debian_version}."',
-        shell=True,
-    )
-    subprocess.run('dch -M -r -D stable ""', shell=True)
+    # Generate debian changelogs if its not an RC.
+    if not rc:
+        subprocess.run(
+            f'dch -M -v {new_version} "New synapse release {new_version}."', shell=True
+        )
+        subprocess.run('dch -M -r -D stable ""', shell=True)

    # Show the user the changes and ask if they want to edit the change log.
    repo.git.add("-u")
@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.39.0"
+__version__ = "1.39.0rc2"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -120,6 +120,7 @@ class EventTypes:
    SpaceParent = "m.space.parent"

    MSC2716_INSERTION = "org.matrix.msc2716.insertion"
+    MSC2716_CHUNK = "org.matrix.msc2716.chunk"
    MSC2716_MARKER = "org.matrix.msc2716.marker"


@@ -127,14 +128,6 @@ class ToDeviceEventTypes:
    RoomKeyRequest = "m.room_key_request"


-class DeviceKeyAlgorithms:
-    """Spec'd algorithms for the generation of per-device keys"""
-
-    ED25519 = "ed25519"
-    CURVE25519 = "curve25519"
-    SIGNED_CURVE25519 = "signed_curve25519"
-
-
 class EduTypes:
    Presence = "m.presence"

@@ -198,9 +191,10 @@ class EventContentFields:

    # Used on normal messages to indicate they were historically imported after the fact
    MSC2716_HISTORICAL = "org.matrix.msc2716.historical"
-    # For "insertion" events
+    # For "insertion" events to indicate what the next chunk ID should be in
+    # order to connect to it
    MSC2716_NEXT_CHUNK_ID = "org.matrix.msc2716.next_chunk_id"
-    # Used on normal message events to indicate where the chunk connects to
+    # Used on "chunk" events to indicate which insertion event it connects to
    MSC2716_CHUNK_ID = "org.matrix.msc2716.chunk_id"
    # For "marker" events
    MSC2716_MARKER_INSERTION = "org.matrix.msc2716.marker.insertion"
@@ -75,6 +75,9 @@ class Codes:
    INVALID_SIGNATURE = "M_INVALID_SIGNATURE"
    USER_DEACTIVATED = "M_USER_DEACTIVATED"
    BAD_ALIAS = "M_BAD_ALIAS"
+    # For restricted join rules.
+    UNABLE_AUTHORISE_JOIN = "M_UNABLE_TO_AUTHORISE_JOIN"
+    UNABLE_TO_GRANT_JOIN = "M_UNABLE_TO_GRANT_JOIN"


 class CodeMessageException(RuntimeError):
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from typing import Dict
+from typing import Callable, Dict, Optional

 import attr

@@ -168,7 +168,7 @@ class RoomVersions:
        msc2403_knocking=False,
    )
    MSC3083 = RoomVersion(
-        "org.matrix.msc3083",
+        "org.matrix.msc3083.v2",
        RoomDisposition.UNSTABLE,
        EventFormatVersions.V3,
        StateResolutionVersions.V2,
@@ -208,5 +208,39 @@ KNOWN_ROOM_VERSIONS: Dict[str, RoomVersion] = {
        RoomVersions.MSC3083,
        RoomVersions.V7,
    )
-    # Note that we do not include MSC2043 here unless it is enabled in the config.
+}
+
+
+@attr.s(slots=True, frozen=True, auto_attribs=True)
+class RoomVersionCapability:
+    """An object which describes the unique attributes of a room version."""
+
+    identifier: str  # the identifier for this capability
+    preferred_version: Optional[RoomVersion]
+    support_check_lambda: Callable[[RoomVersion], bool]
+
+
+MSC3244_CAPABILITIES = {
+    cap.identifier: {
+        "preferred": cap.preferred_version.identifier
+        if cap.preferred_version is not None
+        else None,
+        "support": [
+            v.identifier
+            for v in KNOWN_ROOM_VERSIONS.values()
+            if cap.support_check_lambda(v)
+        ],
+    }
+    for cap in (
+        RoomVersionCapability(
+            "knock",
+            RoomVersions.V7,
+            lambda room_version: room_version.msc2403_knocking,
+        ),
+        RoomVersionCapability(
+            "restricted",
+            None,
+            lambda room_version: room_version.msc3083_join_rules,
+        ),
+    )
 }
@@ -109,7 +109,7 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
    for name, count in r30_results.items():
        stats["r30_users_" + name] = count

-    r30v2_results = await store.count_r30v2_users()
+    r30v2_results = await store.count_r30_users()
    for name, count in r30v2_results.items():
        stats["r30v2_users_" + name] = count

@@ -39,12 +39,13 @@ DEFAULT_SUBJECTS = {
    "messages_from_person_and_others": "[%(app)s] You have messages on %(app)s from %(person)s and others...",
    "invite_from_person": "[%(app)s] %(person)s has invited you to chat on %(app)s...",
    "invite_from_person_to_room": "[%(app)s] %(person)s has invited you to join the %(room)s room on %(app)s...",
+    "invite_from_person_to_space": "[%(app)s] %(person)s has invited you to join the %(space)s space on %(app)s...",
    "password_reset": "[%(server_name)s] Password reset",
    "email_validation": "[%(server_name)s] Validate your email",
 }


-@attr.s
+@attr.s(slots=True, frozen=True)
 class EmailSubjectConfig:
    message_from_person_in_room = attr.ib(type=str)
    message_from_person = attr.ib(type=str)
@@ -54,6 +55,7 @@ class EmailSubjectConfig:
    messages_from_person_and_others = attr.ib(type=str)
    invite_from_person = attr.ib(type=str)
    invite_from_person_to_room = attr.ib(type=str)
+    invite_from_person_to_space = attr.ib(type=str)
    password_reset = attr.ib(type=str)
    email_validation = attr.ib(type=str)

@@ -32,3 +32,6 @@ class ExperimentalConfig(Config):

        # MSC2716 (backfill existing history)
        self.msc2716_enabled: bool = experimental.get("msc2716_enabled", False)
+
+        # MSC3244 (room version capabilities)
+        self.msc3244_enabled: bool = experimental.get("msc3244_enabled", False)
@@ -71,7 +71,7 @@ handlers:
    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
    # logs will still be flushed immediately.
    buffer:
-        class: logging.handlers.MemoryHandler
+        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
        target: file
        # The capacity is the number of log lines that are buffered before
        # being written to disk. Increasing this will lead to better
@@ -79,6 +79,9 @@ handlers:
        # be written to disk.
        capacity: 10
        flushLevel: 30  # Flush for WARNING logs as well
+        # The period of time, in seconds, between forced flushes.
+        # Messages will not be delayed for longer than this time.
+        period: 5

    # A handler that writes logs to stderr. Unused by default, but can be used
    # instead of "buffer" and "file" in the logger handlers.
@@ -106,6 +106,18 @@ def check(
            if not event.signatures.get(event_id_domain):
                raise AuthError(403, "Event not signed by sending server")

+        is_invite_via_allow_rule = (
+            event.type == EventTypes.Member
+            and event.membership == Membership.JOIN
+            and "join_authorised_via_users_server" in event.content
+        )
+        if is_invite_via_allow_rule:
+            authoriser_domain = get_domain_from_id(
+                event.content["join_authorised_via_users_server"]
+            )
+            if not event.signatures.get(authoriser_domain):
+                raise AuthError(403, "Event not signed by authorising server")
+
    # Implementation of https://matrix.org/docs/spec/rooms/v1#authorization-rules
    #
    # 1. If type is m.room.create:
@@ -177,7 +189,7 @@ def check(
    # https://github.com/vector-im/vector-web/issues/1208 hopefully
    if event.type == EventTypes.ThirdPartyInvite:
        user_level = get_user_power_level(event.user_id, auth_events)
-        invite_level = _get_named_level(auth_events, "invite", 0)
+        invite_level = get_named_level(auth_events, "invite", 0)

        if user_level < invite_level:
            raise AuthError(403, "You don't have permission to invite users")
@@ -285,8 +297,8 @@ def _is_membership_change_allowed(
    user_level = get_user_power_level(event.user_id, auth_events)
    target_level = get_user_power_level(target_user_id, auth_events)

-    # FIXME (erikj): What should we do here as the default?
-    ban_level = _get_named_level(auth_events, "ban", 50)
+    invite_level = get_named_level(auth_events, "invite", 0)
+    ban_level = get_named_level(auth_events, "ban", 50)

    logger.debug(
        "_is_membership_change_allowed: %s",
@@ -336,8 +348,6 @@ def _is_membership_change_allowed(
        elif target_in_room:  # the target is already in the room.
            raise AuthError(403, "%s is already in the room." % target_user_id)
        else:
-            invite_level = _get_named_level(auth_events, "invite", 0)
-
            if user_level < invite_level:
                raise AuthError(403, "You don't have permission to invite users")
    elif Membership.JOIN == membership:
@@ -345,16 +355,41 @@ def _is_membership_change_allowed(
        # * They are not banned.
        # * They are accepting a previously sent invitation.
        # * They are already joined (it's a NOOP).
-        # * The room is public or restricted.
+        # * The room is public.
+        # * The room is restricted and the user meets the allows rules.
        if event.user_id != target_user_id:
            raise AuthError(403, "Cannot force another user to join.")
        elif target_banned:
            raise AuthError(403, "You are banned from this room")
-        elif join_rule == JoinRules.PUBLIC or (
+        elif join_rule == JoinRules.PUBLIC:
+            pass
+        elif (
            room_version.msc3083_join_rules
            and join_rule == JoinRules.MSC3083_RESTRICTED
        ):
-            pass
+            # This is the same as public, but the event must contain a reference
+            # to the server who authorised the join. If the event does not contain
+            # the proper content it is rejected.
+            #
+            # Note that if the caller is in the room or invited, then they do
+            # not need to meet the allow rules.
+            if not caller_in_room and not caller_invited:
+                authorising_user = event.content.get("join_authorised_via_users_server")
+
+                if authorising_user is None:
+                    raise AuthError(403, "Join event is missing authorising user.")
+
+                # The authorising user must be in the room.
+                key = (EventTypes.Member, authorising_user)
+                member_event = auth_events.get(key)
+                _check_joined_room(member_event, authorising_user, event.room_id)
+
+                authorising_user_level = get_user_power_level(
+                    authorising_user, auth_events
+                )
+                if authorising_user_level < invite_level:
+                    raise AuthError(403, "Join event authorised by invalid server.")
+
        elif join_rule == JoinRules.INVITE or (
            room_version.msc2403_knocking and join_rule == JoinRules.KNOCK
        ):
@@ -369,7 +404,7 @@ def _is_membership_change_allowed(
        if target_banned and user_level < ban_level:
            raise AuthError(403, "You cannot unban user %s." % (target_user_id,))
        elif target_user_id != event.user_id:
-            kick_level = _get_named_level(auth_events, "kick", 50)
+            kick_level = get_named_level(auth_events, "kick", 50)

            if user_level < kick_level or user_level <= target_level:
                raise AuthError(403, "You cannot kick user %s." % target_user_id)
@@ -445,7 +480,7 @@ def get_send_level(


 def _can_send_event(event: EventBase, auth_events: StateMap[EventBase]) -> bool:
-    power_levels_event = _get_power_level_event(auth_events)
+    power_levels_event = get_power_level_event(auth_events)

    send_level = get_send_level(event.type, event.get("state_key"), power_levels_event)
    user_level = get_user_power_level(event.user_id, auth_events)
@@ -485,7 +520,7 @@ def check_redaction(
    """
    user_level = get_user_power_level(event.user_id, auth_events)

-    redact_level = _get_named_level(auth_events, "redact", 50)
+    redact_level = get_named_level(auth_events, "redact", 50)

    if user_level >= redact_level:
        return False
@@ -600,7 +635,7 @@ def _check_power_levels(
            )


-def _get_power_level_event(auth_events: StateMap[EventBase]) -> Optional[EventBase]:
+def get_power_level_event(auth_events: StateMap[EventBase]) -> Optional[EventBase]:
    return auth_events.get((EventTypes.PowerLevels, ""))


@@ -616,7 +651,7 @@ def get_user_power_level(user_id: str, auth_events: StateMap[EventBase]) -> int:
    Returns:
        the user's power level in this room.
    """
-    power_level_event = _get_power_level_event(auth_events)
+    power_level_event = get_power_level_event(auth_events)
    if power_level_event:
        level = power_level_event.content.get("users", {}).get(user_id)
        if not level:
@@ -640,8 +675,8 @@ def get_user_power_level(user_id: str, auth_events: StateMap[EventBase]) -> int:
            return 0


-def _get_named_level(auth_events: StateMap[EventBase], name: str, default: int) -> int:
-    power_level_event = _get_power_level_event(auth_events)
+def get_named_level(auth_events: StateMap[EventBase], name: str, default: int) -> int:
+    power_level_event = get_power_level_event(auth_events)

    if not power_level_event:
        return default
@@ -728,7 +763,9 @@ def get_public_keys(invite_event: EventBase) -> List[Dict[str, Any]]:
    return public_keys


-def auth_types_for_event(event: Union[EventBase, EventBuilder]) -> Set[Tuple[str, str]]:
+def auth_types_for_event(
+    room_version: RoomVersion, event: Union[EventBase, EventBuilder]
+) -> Set[Tuple[str, str]]:
    """Given an event, return a list of (EventType, StateKey) that may be
    needed to auth the event. The returned list may be a superset of what
    would actually be required depending on the full state of the room.
@@ -760,4 +797,12 @@ def auth_types_for_event(event: Union[EventBase, EventBuilder]) -> Set[Tuple[str
                )
                auth_types.add(key)

+        if room_version.msc3083_join_rules and membership == Membership.JOIN:
+            if "join_authorised_via_users_server" in event.content:
+                key = (
+                    EventTypes.Member,
+                    event.content["join_authorised_via_users_server"],
+                )
+                auth_types.add(key)
+
    return auth_types
@@ -178,6 +178,34 @@ async def _check_sigs_on_pdu(
            )
            raise SynapseError(403, errmsg, Codes.FORBIDDEN)

+    # If this is a join event for a restricted room it may have been authorised
+    # via a different server from the sending server. Check those signatures.
+    if (
+        room_version.msc3083_join_rules
+        and pdu.type == EventTypes.Member
+        and pdu.membership == Membership.JOIN
+        and "join_authorised_via_users_server" in pdu.content
+    ):
+        authorising_server = get_domain_from_id(
+            pdu.content["join_authorised_via_users_server"]
+        )
+        try:
+            await keyring.verify_event_for_server(
+                authorising_server,
+                pdu,
+                pdu.origin_server_ts if room_version.enforce_key_validity else 0,
+            )
+        except Exception as e:
+            errmsg = (
+                "event id %s: unable to verify signature for authorising server %s: %s"
+                % (
+                    pdu.event_id,
+                    authorising_server,
+                    e,
+                )
+            )
+            raise SynapseError(403, errmsg, Codes.FORBIDDEN)
+

 def _is_invite_via_3pid(event: EventBase) -> bool:
    return (
@@ -19,7 +19,6 @@ import itertools
 import logging
 from typing import (
    TYPE_CHECKING,
-    Any,
    Awaitable,
    Callable,
    Collection,
@@ -79,7 +78,15 @@ class InvalidResponseError(RuntimeError):
    we couldn't parse
    """

-    pass
+
+@attr.s(slots=True, frozen=True, auto_attribs=True)
+class SendJoinResult:
+    # The event to persist.
+    event: EventBase
+    # A string giving the server the event was sent to.
+    origin: str
+    state: List[EventBase]
+    auth_chain: List[EventBase]


 class FederationClient(FederationBase):
@@ -677,7 +684,7 @@ class FederationClient(FederationBase):

    async def send_join(
        self, destinations: Iterable[str], pdu: EventBase, room_version: RoomVersion
-    ) -> Dict[str, Any]:
+    ) -> SendJoinResult:
        """Sends a join event to one of a list of homeservers.

        Doing so will cause the remote server to add the event to the graph,
@@ -691,18 +698,38 @@ class FederationClient(FederationBase):
                did the make_join)

        Returns:
-            a dict with members ``origin`` (a string
-            giving the server the event was sent to, ``state`` (?) and
-            ``auth_chain``.
+            The result of the send join request.

        Raises:
            SynapseError: if the chosen remote server returns a 300/400 code, or
                no servers successfully handle the request.
        """

-        async def send_request(destination) -> Dict[str, Any]:
+        async def send_request(destination) -> SendJoinResult:
            response = await self._do_send_join(room_version, destination, pdu)

+            # If an event was returned (and expected to be returned):
+            #
+            # * Ensure it has the same event ID (note that the event ID is a hash
+            #   of the event fields for versions which support MSC3083).
+            # * Ensure the signatures are good.
+            #
+            # Otherwise, fallback to the provided event.
+            if room_version.msc3083_join_rules and response.event:
+                event = response.event
+
+                valid_pdu = await self._check_sigs_and_hash_and_fetch_one(
+                    pdu=event,
+                    origin=destination,
+                    outlier=True,
+                    room_version=room_version,
+                )
+
+                if valid_pdu is None or event.event_id != pdu.event_id:
+                    raise InvalidResponseError("Returned an invalid join event")
+            else:
+                event = pdu
+
            state = response.state
            auth_chain = response.auth_events

@@ -784,11 +811,21 @@ class FederationClient(FederationBase):
                    % (auth_chain_create_events,)
                )

-            return {
-                "state": signed_state,
-                "auth_chain": signed_auth,
-                "origin": destination,
-            }
+            return SendJoinResult(
+                event=event,
+                state=signed_state,
+                auth_chain=signed_auth,
+                origin=destination,
+            )
+
+        if room_version.msc3083_join_rules:
+            # If the join is being authorised via allow rules, we need to send
+            # the /send_join back to the same server that was originally used
+            # with /make_join.
+            if "join_authorised_via_users_server" in pdu.content:
+                destinations = [
+                    get_domain_from_id(pdu.content["join_authorised_via_users_server"])
+                ]

        return await self._try_destination_list("send_join", destinations, send_request)

@@ -45,6 +45,7 @@ from synapse.api.errors import (
    UnsupportedRoomVersionError,
 )
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
+from synapse.crypto.event_signing import compute_event_signature
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.federation.federation_base import FederationBase, event_from_pdu_json
@@ -64,7 +65,7 @@ from synapse.replication.http.federation import (
    ReplicationGetQueryRestServlet,
 )
 from synapse.storage.databases.main.lock import Lock
-from synapse.types import JsonDict
+from synapse.types import JsonDict, get_domain_from_id
 from synapse.util import glob_to_regex, json_decoder, unwrapFirstError
 from synapse.util.async_helpers import Linearizer, concurrently_execute
 from synapse.util.caches.response_cache import ResponseCache
@@ -586,7 +587,7 @@ class FederationServer(FederationBase):
    async def on_send_join_request(
        self, origin: str, content: JsonDict, room_id: str
    ) -> Dict[str, Any]:
-        context = await self._on_send_membership_event(
+        event, context = await self._on_send_membership_event(
            origin, content, Membership.JOIN, room_id
        )

@@ -597,6 +598,7 @@ class FederationServer(FederationBase):

        time_now = self._clock.time_msec()
        return {
+            "org.matrix.msc3083.v2.event": event.get_pdu_json(),
            "state": [p.get_pdu_json(time_now) for p in state.values()],
            "auth_chain": [p.get_pdu_json(time_now) for p in auth_chain],
        }
@@ -681,7 +683,7 @@ class FederationServer(FederationBase):
        Returns:
            The stripped room state.
        """
-        event_context = await self._on_send_membership_event(
+        _, context = await self._on_send_membership_event(
            origin, content, Membership.KNOCK, room_id
        )

@@ -690,14 +692,14 @@ class FederationServer(FederationBase):
        # related to the room while the knock request is pending.
        stripped_room_state = (
            await self.store.get_stripped_room_state_from_event_context(
-                event_context, self._room_prejoin_state_types
+                context, self._room_prejoin_state_types
            )
        )
        return {"knock_state_events": stripped_room_state}

    async def _on_send_membership_event(
        self, origin: str, content: JsonDict, membership_type: str, room_id: str
-    ) -> EventContext:
+    ) -> Tuple[EventBase, EventContext]:
        """Handle an on_send_{join,leave,knock} request

        Does some preliminary validation before passing the request on to the
@@ -712,7 +714,7 @@ class FederationServer(FederationBase):
                in the event

        Returns:
-            The context of the event after inserting it into the room graph.
+            The event and context of the event after inserting it into the room graph.

        Raises:
            SynapseError if there is a problem with the request, including things like
@@ -748,6 +750,33 @@ class FederationServer(FederationBase):

        logger.debug("_on_send_membership_event: pdu sigs: %s", event.signatures)

+        # Sign the event since we're vouching on behalf of the remote server that
+        # the event is valid to be sent into the room. Currently this is only done
+        # if the user is being joined via restricted join rules.
+        if (
+            room_version.msc3083_join_rules
+            and event.membership == Membership.JOIN
+            and "join_authorised_via_users_server" in event.content
+        ):
+            # We can only authorise our own users.
+            authorising_server = get_domain_from_id(
+                event.content["join_authorised_via_users_server"]
+            )
+            if authorising_server != self.server_name:
+                raise SynapseError(
+                    400,
+                    f"Cannot authorise request from resident server: {authorising_server}",
+                )
+
+            event.signatures.update(
+                compute_event_signature(
+                    room_version,
+                    event.get_pdu_json(),
+                    self.hs.hostname,
+                    self.hs.signing_key,
+                )
+            )
+
        event = await self._check_sigs_and_hash(room_version, event)

        return await self.handler.on_send_membership_event(origin, event)
@@ -984,7 +984,7 @@ class PublicRoomList(BaseFederationServlet):
        limit = parse_integer_from_args(query, "limit", 0)
        since_token = parse_string_from_args(query, "since", None)
        include_all_networks = parse_boolean_from_args(
-            query, "include_all_networks", False
+            query, "include_all_networks", default=False
        )
        third_party_instance_id = parse_string_from_args(
            query, "third_party_instance_id", None
@@ -1908,16 +1908,7 @@ class FederationSpaceSummaryServlet(BaseFederationServlet):
        suggested_only = parse_boolean_from_args(query, "suggested_only", default=False)
        max_rooms_per_space = parse_integer_from_args(query, "max_rooms_per_space")

-        exclude_rooms = []
-        if b"exclude_rooms" in query:
-            try:
-                exclude_rooms = [
-                    room_id.decode("ascii") for room_id in query[b"exclude_rooms"]
-                ]
-            except Exception:
-                raise SynapseError(
-                    400, "Bad query parameter for exclude_rooms", Codes.INVALID_PARAM
-                )
+        exclude_rooms = parse_strings_from_args(query, "exclude_rooms", default=[])

        return 200, await self.handler.federation_space_summary(
            origin, room_id, suggested_only, max_rooms_per_space, exclude_rooms
@@ -15,6 +15,8 @@
 import logging
 from typing import TYPE_CHECKING, Optional

+import synapse.state
+import synapse.storage
 import synapse.types
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.ratelimiting import Ratelimiter
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import logging
 from typing import TYPE_CHECKING, Collection, List, Optional, Union

 from synapse import event_auth
@@ -20,16 +21,18 @@ from synapse.api.constants import (
    Membership,
    RestrictedJoinRuleTypes,
 )
-from synapse.api.errors import AuthError
+from synapse.api.errors import AuthError, Codes, SynapseError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
 from synapse.events import EventBase
 from synapse.events.builder import EventBuilder
-from synapse.types import StateMap
+from synapse.types import StateMap, get_domain_from_id
 from synapse.util.metrics import Measure

 if TYPE_CHECKING:
    from synapse.server import HomeServer

+logger = logging.getLogger(__name__)
+

 class EventAuthHandler:
    """
@@ -39,6 +42,7 @@ class EventAuthHandler:
    def __init__(self, hs: "HomeServer"):
        self._clock = hs.get_clock()
        self._store = hs.get_datastore()
+        self._server_name = hs.hostname

    async def check_from_context(
        self, room_version: str, event, context, do_sig_check=True
@@ -81,15 +85,76 @@ class EventAuthHandler:
        # introduce undesirable "state reset" behaviour.
        #
        # All of which sounds a bit tricky so we don't bother for now.
-
        auth_ids = []
-        for etype, state_key in event_auth.auth_types_for_event(event):
+        for etype, state_key in event_auth.auth_types_for_event(
+            event.room_version, event
+        ):
            auth_ev_id = current_state_ids.get((etype, state_key))
            if auth_ev_id:
                auth_ids.append(auth_ev_id)

        return auth_ids

+    async def get_user_which_could_invite(
+        self, room_id: str, current_state_ids: StateMap[str]
+    ) -> str:
+        """
+        Searches the room state for a local user who has the power level necessary
+        to invite other users.
+
+        Args:
+            room_id: The room ID under search.
+            current_state_ids: The current state of the room.
+
+        Returns:
+            The MXID of the user which could issue an invite.
+
+        Raises:
+            SynapseError if no appropriate user is found.
+        """
+        power_level_event_id = current_state_ids.get((EventTypes.PowerLevels, ""))
+        invite_level = 0
+        users_default_level = 0
+        if power_level_event_id:
+            power_level_event = await self._store.get_event(power_level_event_id)
+            invite_level = power_level_event.content.get("invite", invite_level)
+            users_default_level = power_level_event.content.get(
+                "users_default", users_default_level
+            )
+            users = power_level_event.content.get("users", {})
+        else:
+            users = {}
+
+        # Find the user with the highest power level.
+        users_in_room = await self._store.get_users_in_room(room_id)
+        # Only interested in local users.
+        local_users_in_room = [
+            u for u in users_in_room if get_domain_from_id(u) == self._server_name
+        ]
+        chosen_user = max(
+            local_users_in_room,
+            key=lambda user: users.get(user, users_default_level),
+            default=None,
+        )
+
+        # Return the chosen if they can issue invites.
+        user_power_level = users.get(chosen_user, users_default_level)
+        if chosen_user and user_power_level >= invite_level:
+            logger.debug(
+                "Found a user who can issue invites  %s with power level %d >= invite level %d",
+                chosen_user,
+                user_power_level,
+                invite_level,
+            )
+            return chosen_user
+
+        # No user was found.
+        raise SynapseError(
+            400,
+            "Unable to find a user which could issue an invite",
+            Codes.UNABLE_TO_GRANT_JOIN,
+        )
+
    async def check_host_in_room(self, room_id: str, host: str) -> bool:
        with Measure(self._clock, "check_host_in_room"):
            return await self._store.is_host_joined(room_id, host)
@@ -134,6 +199,18 @@ class EventAuthHandler:
        # in any of them.
        allowed_rooms = await self.get_rooms_that_allow_join(state_ids)
        if not await self.is_user_in_rooms(allowed_rooms, user_id):
+
+            # If this is a remote request, the user might be in an allowed room
+            # that we do not know about.
+            if get_domain_from_id(user_id) != self._server_name:
+                for room_id in allowed_rooms:
+                    if not await self._store.is_host_joined(room_id, self._server_name):
+                        raise SynapseError(
+                            400,
+                            f"Unable to check if {user_id} is in allowed rooms.",
+                            Codes.UNABLE_AUTHORISE_JOIN,
+                        )
+
            raise AuthError(
                403,
                "You do not belong to any of the required rooms to join this room.",
@@ -1494,9 +1494,10 @@ class FederationHandler(BaseHandler):
                host_list, event, room_version_obj
            )

-            origin = ret["origin"]
-            state = ret["state"]
-            auth_chain = ret["auth_chain"]
+            event = ret.event
+            origin = ret.origin
+            state = ret.state
+            auth_chain = ret.auth_chain
            auth_chain.sort(key=lambda e: e.depth)

            logger.debug("do_invite_join auth_chain: %s", auth_chain)
@@ -1676,7 +1677,7 @@ class FederationHandler(BaseHandler):

        # checking the room version will check that we've actually heard of the room
        # (and return a 404 otherwise)
-        room_version = await self.store.get_room_version_id(room_id)
+        room_version = await self.store.get_room_version(room_id)

        # now check that we are *still* in the room
        is_in_room = await self._event_auth_handler.check_host_in_room(
@@ -1691,8 +1692,38 @@ class FederationHandler(BaseHandler):

        event_content = {"membership": Membership.JOIN}

+        # If the current room is using restricted join rules, additional information
+        # may need to be included in the event content in order to efficiently
+        # validate the event.
+        #
+        # Note that this requires the /send_join request to come back to the
+        # same server.
+        if room_version.msc3083_join_rules:
+            state_ids = await self.store.get_current_state_ids(room_id)
+            if await self._event_auth_handler.has_restricted_join_rules(
+                state_ids, room_version
+            ):
+                prev_member_event_id = state_ids.get((EventTypes.Member, user_id), None)
+                # If the user is invited or joined to the room already, then
+                # no additional info is needed.
+                include_auth_user_id = True
+                if prev_member_event_id:
+                    prev_member_event = await self.store.get_event(prev_member_event_id)
+                    include_auth_user_id = prev_member_event.membership not in (
+                        Membership.JOIN,
+                        Membership.INVITE,
+                    )
+
+                if include_auth_user_id:
+                    event_content[
+                        "join_authorised_via_users_server"
+                    ] = await self._event_auth_handler.get_user_which_could_invite(
+                        room_id,
+                        state_ids,
+                    )
+
        builder = self.event_builder_factory.new(
-            room_version,
+            room_version.identifier,
            {
                "type": EventTypes.Member,
                "content": event_content,
@@ -1710,10 +1741,13 @@ class FederationHandler(BaseHandler):
            logger.warning("Failed to create join to %s because %s", room_id, e)
            raise

+        # Ensure the user can even join the room.
+        await self._check_join_restrictions(context, event)
+
        # The remote hasn't signed it yet, obviously. We'll do the full checks
        # when we get the event back in `on_send_join_request`
        await self._event_auth_handler.check_from_context(
-            room_version, event, context, do_sig_check=False
+            room_version.identifier, event, context, do_sig_check=False
        )

        return event
@@ -1958,7 +1992,7 @@ class FederationHandler(BaseHandler):
    @log_function
    async def on_send_membership_event(
        self, origin: str, event: EventBase
-    ) -> EventContext:
+    ) -> Tuple[EventBase, EventContext]:
        """
        We have received a join/leave/knock event for a room via send_join/leave/knock.

@@ -1981,7 +2015,7 @@ class FederationHandler(BaseHandler):
            event: The member event that has been signed by the remote homeserver.

        Returns:
-            The context of the event after inserting it into the room graph.
+            The event and context of the event after inserting it into the room graph.

        Raises:
            SynapseError if the event is not accepted into the room
@@ -2037,7 +2071,7 @@ class FederationHandler(BaseHandler):

        # all looks good, we can persist the event.
        await self._run_push_actions_and_persist_event(event, context)
-        return context
+        return event, context

    async def _check_join_restrictions(
        self, context: EventContext, event: EventBase
@@ -2473,7 +2507,7 @@ class FederationHandler(BaseHandler):
        )

        # Now check if event pass auth against said current state
-        auth_types = auth_types_for_event(event)
+        auth_types = auth_types_for_event(room_version_obj, event)
        current_state_ids_list = [
            e for k, e in current_state_ids.items() if k in auth_types
        ]
@@ -16,7 +16,7 @@ import abc
 import logging
 import random
 from http import HTTPStatus
-from typing import TYPE_CHECKING, Iterable, List, Optional, Tuple
+from typing import TYPE_CHECKING, Iterable, List, Optional, Set, Tuple

 from synapse import types
 from synapse.api.constants import AccountDataTypes, EventTypes, Membership
@@ -28,6 +28,7 @@ from synapse.api.errors import (
    SynapseError,
 )
 from synapse.api.ratelimiting import Ratelimiter
+from synapse.event_auth import get_named_level, get_power_level_event
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.types import (
@@ -340,16 +341,10 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):

        if event.membership == Membership.JOIN:
            newly_joined = True
-            prev_member_event = None
            if prev_member_event_id:
                prev_member_event = await self.store.get_event(prev_member_event_id)
                newly_joined = prev_member_event.membership != Membership.JOIN

-            # Check if the member should be allowed access via membership in a space.
-            await self.event_auth_handler.check_restricted_join_rules(
-                prev_state_ids, event.room_version, user_id, prev_member_event
-            )
-
            # Only rate-limit if the user actually joined the room, otherwise we'll end
            # up blocking profile updates.
            if newly_joined and ratelimit:
@@ -701,7 +696,11 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                    # so don't really fit into the general auth process.
                    raise AuthError(403, "Guest access not allowed")

-            if not is_host_in_room:
+            # Check if a remote join should be performed.
+            remote_join, remote_room_hosts = await self._should_perform_remote_join(
+                target.to_string(), room_id, remote_room_hosts, content, is_host_in_room
+            )
+            if remote_join:
                if ratelimit:
                    time_now_s = self.clock.time()
                    (
@@ -826,6 +825,106 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
            outlier=outlier,
        )

+    async def _should_perform_remote_join(
+        self,
+        user_id: str,
+        room_id: str,
+        remote_room_hosts: List[str],
+        content: JsonDict,
+        is_host_in_room: bool,
+    ) -> Tuple[bool, List[str]]:
+        """
+        Check whether the server should do a remote join (as opposed to a local
+        join) for a user.
+
+        Generally a remote join is used if:
+
+        * The server is not yet in the room.
+        * The server is in the room, the room has restricted join rules, the user
+          is not joined or invited to the room, and the server does not have
+          another user who is capable of issuing invites.
+
+        Args:
+            user_id: The user joining the room.
+            room_id: The room being joined.
+            remote_room_hosts: A list of remote room hosts.
+            content: The content to use as the event body of the join. This may
+                be modified.
+            is_host_in_room: True if the host is in the room.
+
+        Returns:
+            A tuple of:
+                True if a remote join should be performed. False if the join can be
+                done locally.
+
+                A list of remote room hosts to use. This is an empty list if a
+                local join is to be done.
+        """
+        # If the host isn't in the room, pass through the prospective hosts.
+        if not is_host_in_room:
+            return True, remote_room_hosts
+
+        # If the host is in the room, but not one of the authorised hosts
+        # for restricted join rules, a remote join must be used.
+        room_version = await self.store.get_room_version(room_id)
+        current_state_ids = await self.store.get_current_state_ids(room_id)
+
+        # If restricted join rules are not being used, a local join can always
+        # be used.
+        if not await self.event_auth_handler.has_restricted_join_rules(
+            current_state_ids, room_version
+        ):
+            return False, []
+
+        # If the user is invited to the room or already joined, the join
+        # event can always be issued locally.
+        prev_member_event_id = current_state_ids.get((EventTypes.Member, user_id), None)
+        prev_member_event = None
+        if prev_member_event_id:
+            prev_member_event = await self.store.get_event(prev_member_event_id)
+            if prev_member_event.membership in (
+                Membership.JOIN,
+                Membership.INVITE,
+            ):
+                return False, []
+
+        # If the local host has a user who can issue invites, then a local
+        # join can be done.
+        #
+        # If not, generate a new list of remote hosts based on which
+        # can issue invites.
+        event_map = await self.store.get_events(current_state_ids.values())
+        current_state = {
+            state_key: event_map[event_id]
+            for state_key, event_id in current_state_ids.items()
+        }
+        allowed_servers = get_servers_from_users(
+            get_users_which_can_issue_invite(current_state)
+        )
+
+        # If the local server is not one of allowed servers, then a remote
+        # join must be done. Return the list of prospective servers based on
+        # which can issue invites.
+        if self.hs.hostname not in allowed_servers:
+            return True, list(allowed_servers)
+
+        # Ensure the member should be allowed access via membership in a room.
+        await self.event_auth_handler.check_restricted_join_rules(
+            current_state_ids, room_version, user_id, prev_member_event
+        )
+
+        # If this is going to be a local join, additional information must
+        # be included in the event content in order to efficiently validate
+        # the event.
+        content[
+            "join_authorised_via_users_server"
+        ] = await self.event_auth_handler.get_user_which_could_invite(
+            room_id,
+            current_state_ids,
+        )
+
+        return False, []
+
    async def transfer_room_state_on_room_upgrade(
        self, old_room_id: str, room_id: str
    ) -> None:
@@ -1514,3 +1613,63 @@ class RoomMemberMasterHandler(RoomMemberHandler):

        if membership:
            await self.store.forget(user_id, room_id)
+
+
+def get_users_which_can_issue_invite(auth_events: StateMap[EventBase]) -> List[str]:
+    """
+    Return the list of users which can issue invites.
+
+    This is done by exploring the joined users and comparing their power levels
+    to the necessyar power level to issue an invite.
+
+    Args:
+        auth_events: state in force at this point in the room
+
+    Returns:
+        The users which can issue invites.
+    """
+    invite_level = get_named_level(auth_events, "invite", 0)
+    users_default_level = get_named_level(auth_events, "users_default", 0)
+    power_level_event = get_power_level_event(auth_events)
+
+    # Custom power-levels for users.
+    if power_level_event:
+        users = power_level_event.content.get("users", {})
+    else:
+        users = {}
+
+    result = []
+
+    # Check which members are able to invite by ensuring they're joined and have
+    # the necessary power level.
+    for (event_type, state_key), event in auth_events.items():
+        if event_type != EventTypes.Member:
+            continue
+
+        if event.membership != Membership.JOIN:
+            continue
+
+        # Check if the user has a custom power level.
+        if users.get(state_key, users_default_level) >= invite_level:
+            result.append(state_key)
+
+    return result
+
+
+def get_servers_from_users(users: List[str]) -> Set[str]:
+    """
+    Resolve a list of users into their servers.
+
+    Args:
+        users: A list of users.
+
+    Returns:
+        A set of servers.
+    """
+    servers = set()
+    for user in users:
+        try:
+            servers.add(get_domain_from_id(user))
+        except SynapseError:
+            pass
+    return servers
@@ -1093,10 +1093,6 @@ class SyncHandler:
        one_time_key_counts: JsonDict = {}
        unused_fallback_key_types: List[str] = []
        if device_id:
-            # TODO: We should have a way to let clients differentiate between the states of:
-            #   * no change in OTK count since the provided since token
-            #   * the server has zero OTKs left for this device
-            #  Spec issue: https://github.com/matrix-org/matrix-doc/issues/3298
            one_time_key_counts = await self.store.count_e2e_one_time_keys(
                user_id, device_id
            )
@@ -22,6 +22,7 @@ from synapse.metrics.background_process_metrics import (
    run_as_background_process,
    wrap_as_background_process,
 )
+from synapse.replication.http.typing import ReplicationTypingRestServlet
 from synapse.replication.tcp.streams import TypingStream
 from synapse.types import JsonDict, Requester, UserID, get_domain_from_id
 from synapse.util.caches.stream_change_cache import StreamChangeCache
@@ -61,7 +62,9 @@ class FollowerTypingHandler:
        if hs.should_send_federation():
            self.federation = hs.get_federation_sender()

-        if hs.config.worker.writers.typing != hs.get_instance_name():
+        self._typing_repl_client = ReplicationTypingRestServlet.make_client(hs)
+        self._typing_worker = hs.config.worker.writers.typing
+        if self._typing_worker != hs.get_instance_name():
            hs.get_federation_registry().register_instance_for_edu(
                "m.typing",
                hs.config.worker.writers.typing,
@@ -199,6 +202,30 @@ class FollowerTypingHandler:
    def get_current_token(self) -> int:
        return self._latest_room_serial

+    async def started_typing(
+        self, target_user: UserID, requester: Requester, room_id: str, timeout: int
+    ) -> None:
+        await self._typing_repl_client(
+            typing=True,
+            instance_name=self._typing_worker,
+            user_id=target_user.to_string(),
+            requester=requester,
+            room_id=room_id,
+            timeout=timeout,
+        )
+
+    async def stopped_typing(
+        self, target_user: UserID, requester: Requester, room_id: str
+    ) -> None:
+        await self._typing_repl_client(
+            typing=True,
+            instance_name=self._typing_worker,
+            user_id=target_user.to_string(),
+            requester=requester,
+            room_id=room_id,
+            timeout=None,
+        )
+

 class TypingWriterHandler(FollowerTypingHandler):
    def __init__(self, hs: "HomeServer"):
@@ -27,7 +27,7 @@ from twisted.internet.interfaces import (
 )
 from twisted.web.client import URI, Agent, HTTPConnectionPool
 from twisted.web.http_headers import Headers
-from twisted.web.iweb import IAgent, IAgentEndpointFactory, IBodyProducer
+from twisted.web.iweb import IAgent, IAgentEndpointFactory, IBodyProducer, IResponse

 from synapse.crypto.context_factory import FederationPolicyForHTTPS
 from synapse.http.client import BlacklistingAgentWrapper
@@ -116,7 +116,7 @@ class MatrixFederationAgent:
        uri: bytes,
        headers: Optional[Headers] = None,
        bodyProducer: Optional[IBodyProducer] = None,
-    ) -> Generator[defer.Deferred, Any, defer.Deferred]:
+    ) -> Generator[defer.Deferred, Any, IResponse]:
        """
        Args:
            method: HTTP method: GET/POST/etc
@@ -14,47 +14,86 @@

 """ This module contains base REST classes for constructing REST servlets. """
 import logging
-from typing import Dict, Iterable, List, Optional, overload
+from typing import Iterable, List, Mapping, Optional, Sequence, overload

 from typing_extensions import Literal

 from twisted.web.server import Request

 from synapse.api.errors import Codes, SynapseError
+from synapse.types import JsonDict
 from synapse.util import json_decoder

 logger = logging.getLogger(__name__)


-def parse_integer(request, name, default=None, required=False):
+@overload
+def parse_integer(request: Request, name: str, default: int) -> int:
+    ...
+
+
+@overload
+def parse_integer(request: Request, name: str, *, required: Literal[True]) -> int:
+    ...
+
+
+@overload
+def parse_integer(
+    request: Request, name: str, default: Optional[int] = None, required: bool = False
+) -> Optional[int]:
+    ...
+
+
+def parse_integer(
+    request: Request, name: str, default: Optional[int] = None, required: bool = False
+) -> Optional[int]:
    """Parse an integer parameter from the request string

    Args:
        request: the twisted HTTP request.
-        name (bytes/unicode): the name of the query parameter.
-        default (int|None): value to use if the parameter is absent, defaults
-            to None.
-        required (bool): whether to raise a 400 SynapseError if the
-            parameter is absent, defaults to False.
+        name: the name of the query parameter.
+        default: value to use if the parameter is absent, defaults to None.
+        required: whether to raise a 400 SynapseError if the parameter is absent,
+            defaults to False.

    Returns:
-        int|None: An int value or the default.
+        An int value or the default.

    Raises:
        SynapseError: if the parameter is absent and required, or if the
            parameter is present and not an integer.
    """
-    return parse_integer_from_args(request.args, name, default, required)
+    args: Mapping[bytes, Sequence[bytes]] = request.args  # type: ignore
+    return parse_integer_from_args(args, name, default, required)


-def parse_integer_from_args(args, name, default=None, required=False):
+def parse_integer_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    default: Optional[int] = None,
+    required: bool = False,
+) -> Optional[int]:
+    """Parse an integer parameter from the request string

-    if not isinstance(name, bytes):
-        name = name.encode("ascii")
+    Args:
+        args: A mapping of request args as bytes to a list of bytes (e.g. request.args).
+        name: the name of the query parameter.
+        default: value to use if the parameter is absent, defaults to None.
+        required: whether to raise a 400 SynapseError if the parameter is absent,
+            defaults to False.

-    if name in args:
+    Returns:
+        An int value or the default.
+
+    Raises:
+        SynapseError: if the parameter is absent and required, or if the
+            parameter is present and not an integer.
+    """
+    name_bytes = name.encode("ascii")
+
+    if name_bytes in args:
        try:
-            return int(args[name][0])
+            return int(args[name_bytes][0])
        except Exception:
            message = "Query parameter %r must be an integer" % (name,)
            raise SynapseError(400, message, errcode=Codes.INVALID_PARAM)
@@ -66,36 +105,102 @@ def parse_integer_from_args(args, name, default=None, required=False):
            return default


-def parse_boolean(request, name, default=None, required=False):
+@overload
+def parse_boolean(request: Request, name: str, default: bool) -> bool:
+    ...
+
+
+@overload
+def parse_boolean(request: Request, name: str, *, required: Literal[True]) -> bool:
+    ...
+
+
+@overload
+def parse_boolean(
+    request: Request, name: str, default: Optional[bool] = None, required: bool = False
+) -> Optional[bool]:
+    ...
+
+
+def parse_boolean(
+    request: Request, name: str, default: Optional[bool] = None, required: bool = False
+) -> Optional[bool]:
    """Parse a boolean parameter from the request query string

    Args:
        request: the twisted HTTP request.
-        name (bytes/unicode): the name of the query parameter.
-        default (bool|None): value to use if the parameter is absent, defaults
-            to None.
-        required (bool): whether to raise a 400 SynapseError if the
-            parameter is absent, defaults to False.
+        name: the name of the query parameter.
+        default: value to use if the parameter is absent, defaults to None.
+        required: whether to raise a 400 SynapseError if the parameter is absent,
+            defaults to False.

    Returns:
-        bool|None: A bool value or the default.
+        A bool value or the default.

    Raises:
        SynapseError: if the parameter is absent and required, or if the
            parameter is present and not one of "true" or "false".
    """
-
-    return parse_boolean_from_args(request.args, name, default, required)
+    args: Mapping[bytes, Sequence[bytes]] = request.args  # type: ignore
+    return parse_boolean_from_args(args, name, default, required)


-def parse_boolean_from_args(args, name, default=None, required=False):
+@overload
+def parse_boolean_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    default: bool,
+) -> bool:
+    ...

-    if not isinstance(name, bytes):
-        name = name.encode("ascii")

-    if name in args:
+@overload
+def parse_boolean_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    *,
+    required: Literal[True],
+) -> bool:
+    ...
+
+
+@overload
+def parse_boolean_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    default: Optional[bool] = None,
+    required: bool = False,
+) -> Optional[bool]:
+    ...
+
+
+def parse_boolean_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    default: Optional[bool] = None,
+    required: bool = False,
+) -> Optional[bool]:
+    """Parse a boolean parameter from the request query string
+
+    Args:
+        args: A mapping of request args as bytes to a list of bytes (e.g. request.args).
+        name: the name of the query parameter.
+        default: value to use if the parameter is absent, defaults to None.
+        required: whether to raise a 400 SynapseError if the parameter is absent,
+            defaults to False.
+
+    Returns:
+        A bool value or the default.
+
+    Raises:
+        SynapseError: if the parameter is absent and required, or if the
+            parameter is present and not one of "true" or "false".
+    """
+    name_bytes = name.encode("ascii")
+
+    if name_bytes in args:
        try:
-            return {b"true": True, b"false": False}[args[name][0]]
+            return {b"true": True, b"false": False}[args[name_bytes][0]]
        except Exception:
            message = (
                "Boolean query parameter %r must be one of ['true', 'false']"
@@ -111,7 +216,7 @@ def parse_boolean_from_args(args, name, default=None, required=False):

@overload
 def parse_bytes_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[bytes] = None,
 ) -> Optional[bytes]:
@@ -120,7 +225,7 @@ def parse_bytes_from_args(

@overload
 def parse_bytes_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Literal[None] = None,
    *,
@@ -131,7 +236,7 @@ def parse_bytes_from_args(

@overload
 def parse_bytes_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[bytes] = None,
    required: bool = False,
@@ -140,7 +245,7 @@ def parse_bytes_from_args(


 def parse_bytes_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[bytes] = None,
    required: bool = False,
@@ -172,6 +277,42 @@ def parse_bytes_from_args(
    return default


+@overload
+def parse_string(
+    request: Request,
+    name: str,
+    default: str,
+    *,
+    allowed_values: Optional[Iterable[str]] = None,
+    encoding: str = "ascii",
+) -> str:
+    ...
+
+
+@overload
+def parse_string(
+    request: Request,
+    name: str,
+    *,
+    required: Literal[True],
+    allowed_values: Optional[Iterable[str]] = None,
+    encoding: str = "ascii",
+) -> str:
+    ...
+
+
+@overload
+def parse_string(
+    request: Request,
+    name: str,
+    *,
+    required: bool = False,
+    allowed_values: Optional[Iterable[str]] = None,
+    encoding: str = "ascii",
+) -> Optional[str]:
+    ...
+
+
 def parse_string(
    request: Request,
    name: str,
@@ -179,7 +320,7 @@ def parse_string(
    required: bool = False,
    allowed_values: Optional[Iterable[str]] = None,
    encoding: str = "ascii",
-):
+) -> Optional[str]:
    """
    Parse a string parameter from the request query string.

@@ -205,7 +346,7 @@ def parse_string(
            parameter is present, must be one of a list of allowed values and
            is not one of those allowed values.
    """
-    args: Dict[bytes, List[bytes]] = request.args  # type: ignore
+    args: Mapping[bytes, Sequence[bytes]] = request.args  # type: ignore
    return parse_string_from_args(
        args,
        name,
@@ -239,9 +380,8 @@ def _parse_string_value(

@overload
 def parse_strings_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
-    default: Optional[List[str]] = None,
    *,
    allowed_values: Optional[Iterable[str]] = None,
    encoding: str = "ascii",
@@ -251,9 +391,20 @@ def parse_strings_from_args(

@overload
 def parse_strings_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
+    name: str,
+    default: List[str],
+    *,
+    allowed_values: Optional[Iterable[str]] = None,
+    encoding: str = "ascii",
+) -> List[str]:
+    ...
+
+
+@overload
+def parse_strings_from_args(
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
-    default: Optional[List[str]] = None,
    *,
    required: Literal[True],
    allowed_values: Optional[Iterable[str]] = None,
@@ -264,7 +415,7 @@ def parse_strings_from_args(

@overload
 def parse_strings_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[List[str]] = None,
    *,
@@ -276,7 +427,7 @@ def parse_strings_from_args(


 def parse_strings_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[List[str]] = None,
    required: bool = False,
@@ -325,7 +476,7 @@ def parse_strings_from_args(

@overload
 def parse_string_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[str] = None,
    *,
@@ -337,7 +488,7 @@ def parse_string_from_args(

@overload
 def parse_string_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[str] = None,
    *,
@@ -350,7 +501,7 @@ def parse_string_from_args(

@overload
 def parse_string_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[str] = None,
    required: bool = False,
@@ -361,7 +512,7 @@ def parse_string_from_args(


 def parse_string_from_args(
-    args: Dict[bytes, List[bytes]],
+    args: Mapping[bytes, Sequence[bytes]],
    name: str,
    default: Optional[str] = None,
    required: bool = False,
@@ -409,13 +560,14 @@ def parse_string_from_args(
    return strings[0]


-def parse_json_value_from_request(request, allow_empty_body=False):
+def parse_json_value_from_request(
+    request: Request, allow_empty_body: bool = False
+) -> Optional[JsonDict]:
    """Parse a JSON value from the body of a twisted HTTP request.

    Args:
        request: the twisted HTTP request.
-        allow_empty_body (bool): if True, an empty body will be accepted and
-            turned into None
+        allow_empty_body: if True, an empty body will be accepted and turned into None

    Returns:
        The JSON value.
@@ -424,7 +576,7 @@ def parse_json_value_from_request(request, allow_empty_body=False):
        SynapseError if the request body couldn't be decoded as JSON.
    """
    try:
-        content_bytes = request.content.read()
+        content_bytes = request.content.read()  # type: ignore
    except Exception:
        raise SynapseError(400, "Error reading JSON content.")

@@ -440,13 +592,15 @@ def parse_json_value_from_request(request, allow_empty_body=False):
    return content


-def parse_json_object_from_request(request, allow_empty_body=False):
+def parse_json_object_from_request(
+    request: Request, allow_empty_body: bool = False
+) -> JsonDict:
    """Parse a JSON object from the body of a twisted HTTP request.

    Args:
        request: the twisted HTTP request.
-        allow_empty_body (bool): if True, an empty body will be accepted and
-            turned into an empty dict.
+        allow_empty_body: if True, an empty body will be accepted and turned into
+            an empty dict.

    Raises:
        SynapseError if the request body couldn't be decoded as JSON or
@@ -457,14 +611,14 @@ def parse_json_object_from_request(request, allow_empty_body=False):
    if allow_empty_body and content is None:
        return {}

-    if type(content) != dict:
+    if not isinstance(content, dict):
        message = "Content must be a JSON object."
        raise SynapseError(400, message, errcode=Codes.BAD_JSON)

    return content


-def assert_params_in_dict(body, required):
+def assert_params_in_dict(body: JsonDict, required: Iterable[str]) -> None:
    absent = []
    for k in required:
        if k not in body:
@@ -25,7 +25,7 @@ See doc/log_contexts.rst for details on how this works.
 import inspect
 import logging
 import threading
-import types
+import typing
 import warnings
 from typing import TYPE_CHECKING, Optional, Tuple, TypeVar, Union

@@ -745,7 +745,7 @@ def run_in_background(f, *args, **kwargs) -> defer.Deferred:
        # by synchronous exceptions, so let's turn them into Failures.
        return defer.fail()

-    if isinstance(res, types.CoroutineType):
+    if isinstance(res, typing.Coroutine):
        res = defer.ensureDeferred(res)

    # At this point we should have a Deferred, if not then f was a synchronous
@@ -0,0 +1,88 @@
+import logging
+import time
+from logging import Handler, LogRecord
+from logging.handlers import MemoryHandler
+from threading import Thread
+from typing import Optional
+
+from twisted.internet.interfaces import IReactorCore
+
+
+class PeriodicallyFlushingMemoryHandler(MemoryHandler):
+    """
+    This is a subclass of MemoryHandler that additionally spawns a background
+    thread to periodically flush the buffer.
+
+    This prevents messages from being buffered for too long.
+
+    Additionally, all messages will be immediately flushed if the reactor has
+    not yet been started.
+    """
+
+    def __init__(
+        self,
+        capacity: int,
+        flushLevel: int = logging.ERROR,
+        target: Optional[Handler] = None,
+        flushOnClose: bool = True,
+        period: float = 5.0,
+        reactor: Optional[IReactorCore] = None,
+    ) -> None:
+        """
+        period: the period between automatic flushes
+
+        reactor: if specified, a custom reactor to use. If not specifies,
+            defaults to the globally-installed reactor.
+            Log entries will be flushed immediately until this reactor has
+            started.
+        """
+        super().__init__(capacity, flushLevel, target, flushOnClose)
+
+        self._flush_period: float = period
+        self._active: bool = True
+        self._reactor_started = False
+
+        self._flushing_thread: Thread = Thread(
+            name="PeriodicallyFlushingMemoryHandler flushing thread",
+            target=self._flush_periodically,
+        )
+        self._flushing_thread.start()
+
+        def on_reactor_running():
+            self._reactor_started = True
+
+        reactor_to_use: IReactorCore
+        if reactor is None:
+            from twisted.internet import reactor as global_reactor
+
+            reactor_to_use = global_reactor  # type: ignore[assignment]
+        else:
+            reactor_to_use = reactor
+
+        # call our hook when the reactor start up
+        reactor_to_use.callWhenRunning(on_reactor_running)
+
+    def shouldFlush(self, record: LogRecord) -> bool:
+        """
+        Before reactor start-up, log everything immediately.
+        Otherwise, fall back to original behaviour of waiting for the buffer to fill.
+        """
+
+        if self._reactor_started:
+            return super().shouldFlush(record)
+        else:
+            return True
+
+    def _flush_periodically(self):
+        """
+        Whilst this handler is active, flush the handler periodically.
+        """
+
+        while self._active:
+            # flush is thread-safe; it acquires and releases the lock internally
+            self.flush()
+            time.sleep(self._flush_period)
+
+    def close(self) -> None:
+        self._active = False
+        super().close()
@@ -484,7 +484,7 @@ class ModuleApi:
    @defer.inlineCallbacks
    def get_state_events_in_room(
        self, room_id: str, types: Iterable[Tuple[str, Optional[str]]]
-    ) -> Generator[defer.Deferred, Any, defer.Deferred]:
+    ) -> Generator[defer.Deferred, Any, Iterable[EventBase]]:
        """Gets current state events for the given room.

        (This is exposed for compatibility with the old SpamCheckerApi. We should
@@ -19,7 +19,7 @@ from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, TypeVar
 import bleach
 import jinja2

-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import EventTypes, Membership, RoomTypes
 from synapse.api.errors import StoreError
 from synapse.config.emailconfig import EmailSubjectConfig
 from synapse.events import EventBase
@@ -600,6 +600,22 @@ class Mailer:
                    "app": self.app_name,
                }

+            # If the room is a space, it gets a slightly different topic.
+            create_event_id = room_state_ids.get(("m.room.create", ""))
+            if create_event_id:
+                create_event = await self.store.get_event(
+                    create_event_id, allow_none=True
+                )
+                if (
+                    create_event
+                    and create_event.content.get("room_type") == RoomTypes.SPACE
+                ):
+                    return self.email_subjects.invite_from_person_to_space % {
+                        "person": inviter_name,
+                        "space": room_name,
+                        "app": self.app_name,
+                    }
+
            return self.email_subjects.invite_from_person_to_room % {
                "person": inviter_name,
                "room": room_name,
@@ -24,6 +24,7 @@ from synapse.replication.http import (
    register,
    send_event,
    streams,
+    typing,
 )

 REPLICATION_PREFIX = "/_synapse/replication"
@@ -43,6 +44,7 @@ class ReplicationRestResource(JsonResource):
        streams.register_servlets(hs, self)
        account_data.register_servlets(hs, self)
        push.register_servlets(hs, self)
+        typing.register_servlets(hs, self)

        # The following can't currently be instantiated on workers.
        if hs.config.worker.worker_app is None:
@@ -0,0 +1,89 @@
+# Copyright 2021 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.types import Requester, UserID
+from typing import TYPE_CHECKING
+import logging
+
+from synapse.http.servlet import parse_json_object_from_request
+from synapse.replication.http._base import ReplicationEndpoint
+
+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
+logger = logging.getLogger(__name__)
+
+
+class ReplicationTypingRestServlet(ReplicationEndpoint):
+    """Call to start or stop a user typing in a room.
+
+    Request format:
+
+        POST /_synapse/replication/typing/:room_id/:user_id
+
+        {
+            "requester": ...,
+            "typing": true,
+            "timeout": 30000
+        }
+
+    """
+
+    NAME = "typing"
+    PATH_ARGS = ("room_id", "user_id")
+    CACHE = False
+
+    def __init__(self, hs: "HomeServer"):
+        super().__init__(hs)
+
+        self.handler = hs.get_typing_handler()
+        self.store = hs.get_datastore()
+
+    @staticmethod
+    async def _serialize_payload(requester, room_id, user_id, typing, timeout):
+        payload = {
+            "requester": requester.serialize(),
+            "typing": typing,
+            "timeout": timeout,
+        }
+
+        return payload
+
+    async def _handle_request(self, request, room_id, user_id):
+        content = parse_json_object_from_request(request)
+
+        requester = Requester.deserialize(self.store, content["requester"])
+        request.requester = requester
+
+        target_user = UserID.from_string(user_id)
+
+        if content["typing"]:
+            await self.handler.started_typing(
+                target_user,
+                requester,
+                room_id,
+                content["timeout"],
+            )
+        else:
+            await self.handler.stopped_typing(
+                target_user,
+                requester,
+                room_id,
+            )
+
+        return 200, {}
+
+
+def register_servlets(hs, http_server):
+    ReplicationTypingRestServlet(hs).register(http_server)
@@ -62,6 +62,7 @@ class UsersRestServletV2(RestServlet):
    The parameter `name` can be used to filter by user id or display name.
    The parameter `guests` can be used to exclude guest users.
    The parameter `deactivated` can be used to include deactivated users.
+    The parameter `order_by` can be used to order the result.
    """

    def __init__(self, hs: "HomeServer"):
@@ -90,8 +91,8 @@ class UsersRestServletV2(RestServlet):
                errcode=Codes.INVALID_PARAM,
            )

-        user_id = parse_string(request, "user_id", default=None)
-        name = parse_string(request, "name", default=None)
+        user_id = parse_string(request, "user_id")
+        name = parse_string(request, "name")
        guests = parse_boolean(request, "guests", default=True)
        deactivated = parse_boolean(request, "deactivated", default=False)

@@ -108,6 +109,7 @@ class UsersRestServletV2(RestServlet):
                UserSortOrder.USER_TYPE.value,
                UserSortOrder.AVATAR_URL.value,
                UserSortOrder.SHADOW_BANNED.value,
+                UserSortOrder.CREATION_TS.value,
            ),
        )

@@ -413,7 +413,7 @@ class RoomBatchSendEventRestServlet(TransactionRestServlet):
        assert_params_in_dict(body, ["state_events_at_start", "events"])

        prev_events_from_query = parse_strings_from_args(request.args, "prev_event")
-        chunk_id_from_query = parse_string(request, "chunk_id", default=None)
+        chunk_id_from_query = parse_string(request, "chunk_id")

        if prev_events_from_query is None:
            raise SynapseError(
@@ -553,9 +553,18 @@ class RoomBatchSendEventRestServlet(TransactionRestServlet):
            ]

        # Connect this current chunk to the insertion event from the previous chunk
-        last_event_in_chunk["content"][
-            EventContentFields.MSC2716_CHUNK_ID
-        ] = chunk_id_to_connect_to
+        chunk_event = {
+            "type": EventTypes.MSC2716_CHUNK,
+            "sender": requester.user.to_string(),
+            "room_id": room_id,
+            "content": {EventContentFields.MSC2716_CHUNK_ID: chunk_id_to_connect_to},
+            # Since the chunk event is put at the end of the chunk,
+            # where the newest-in-time event is, copy the origin_server_ts from
+            # the last event we're inserting
+            "origin_server_ts": last_event_in_chunk["origin_server_ts"],
+        }
+        # Add the chunk event to the end of the chunk (newest-in-time)
+        events_to_create.append(chunk_event)

        # Add an "insertion" event to the start of each chunk (next to the oldest-in-time
        # event in the chunk) so the next chunk can be connected to this one.
@@ -567,7 +576,7 @@ class RoomBatchSendEventRestServlet(TransactionRestServlet):
            # the first event we're inserting
            origin_server_ts=events_to_create[0]["origin_server_ts"],
        )
-        # Prepend the insertion event to the start of the chunk
+        # Prepend the insertion event to the start of the chunk (oldest-in-time)
        events_to_create = [insertion_event] + events_to_create

        event_ids = []
@@ -726,7 +735,7 @@ class PublicRoomListRestServlet(TransactionRestServlet):
        self.auth = hs.get_auth()

    async def on_GET(self, request):
-        server = parse_string(request, "server", default=None)
+        server = parse_string(request, "server")

        try:
            await self.auth.get_user_by_req(request, allow_guest=True)
@@ -745,8 +754,8 @@ class PublicRoomListRestServlet(TransactionRestServlet):
            if server:
                raise e

-        limit = parse_integer(request, "limit", 0)
-        since_token = parse_string(request, "since", None)
+        limit: Optional[int] = parse_integer(request, "limit", 0)
+        since_token = parse_string(request, "since")

        if limit == 0:
            # zero is a special value which corresponds to no limit.
@@ -780,7 +789,7 @@ class PublicRoomListRestServlet(TransactionRestServlet):
    async def on_POST(self, request):
        await self.auth.get_user_by_req(request, allow_guest=True)

-        server = parse_string(request, "server", default=None)
+        server = parse_string(request, "server")
        content = parse_json_object_from_request(request)

        limit: Optional[int] = int(content.get("limit", 100))
@@ -1245,18 +1254,11 @@ class RoomTypingRestServlet(RestServlet):
        self.presence_handler = hs.get_presence_handler()
        self.auth = hs.get_auth()

-        # If we're not on the typing writer instance we should scream if we get
-        # requests.
-        self._is_typing_writer = (
-            hs.config.worker.writers.typing == hs.get_instance_name()
-        )
+        self.handler = hs.get_typing_handler()

    async def on_PUT(self, request, room_id, user_id):
        requester = await self.auth.get_user_by_req(request)

-        if not self._is_typing_writer:
-            raise Exception("Got /typing request on instance that is not typing writer")
-
        room_id = urlparse.unquote(room_id)
        target_user = UserID.from_string(urlparse.unquote(user_id))

@@ -1267,19 +1269,16 @@ class RoomTypingRestServlet(RestServlet):
        # Limit timeout to stop people from setting silly typing timeouts.
        timeout = min(content.get("timeout", 30000), 120000)

-        # Defer getting the typing handler since it will raise on workers.
-        typing_handler = self.hs.get_typing_writer_handler()
-
        try:
            if content["typing"]:
-                await typing_handler.started_typing(
+                await self.handler.started_typing(
                    target_user=target_user,
                    requester=requester,
                    room_id=room_id,
                    timeout=timeout,
                )
            else:
-                await typing_handler.stopped_typing(
+                await self.handler.stopped_typing(
                    target_user=target_user, requester=requester, room_id=room_id
                )
        except ShadowBanError:
@@ -884,7 +884,14 @@ class WhoamiRestServlet(RestServlet):
    async def on_GET(self, request):
        requester = await self.auth.get_user_by_req(request)

-        return 200, {"user_id": requester.user.to_string()}
+        response = {"user_id": requester.user.to_string()}
+
+        # Appservices and similar accounts do not have device IDs
+        # that we can report on, so exclude them for compliance.
+        if requester.device_id is not None:
+            response["device_id"] = requester.device_id
+
+        return 200, response


 def register_servlets(hs, http_server):
@@ -14,7 +14,7 @@
 import logging
 from typing import TYPE_CHECKING, Tuple

-from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
+from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, MSC3244_CAPABILITIES
 from synapse.http.servlet import RestServlet
 from synapse.http.site import SynapseRequest
 from synapse.types import JsonDict
@@ -55,6 +55,12 @@ class CapabilitiesRestServlet(RestServlet):
                "m.change_password": {"enabled": change_password},
            }
        }
+
+        if self.config.experimental.msc3244_enabled:
+            response["capabilities"]["m.room_versions"][
+                "org.matrix.msc3244.room_capabilities"
+            ] = MSC3244_CAPABILITIES
+
        return 200, response


@@ -194,7 +194,7 @@ class KeyChangesServlet(RestServlet):
    async def on_GET(self, request):
        requester = await self.auth.get_user_by_req(request, allow_guest=True)

-        from_token_string = parse_string(request, "from")
+        from_token_string = parse_string(request, "from", required=True)
        set_tag("from", from_token_string)

        # We want to enforce they do pass us one, but we ignore it and return
@@ -158,19 +158,21 @@ class RelationPaginationServlet(RestServlet):
        event = await self.event_handler.get_event(requester.user, room_id, parent_id)

        limit = parse_integer(request, "limit", default=5)
-        from_token = parse_string(request, "from")
-        to_token = parse_string(request, "to")
+        from_token_str = parse_string(request, "from")
+        to_token_str = parse_string(request, "to")

        if event.internal_metadata.is_redacted():
            # If the event is redacted, return an empty list of relations
            pagination_chunk = PaginationChunk(chunk=[])
        else:
            # Return the relations
-            if from_token:
-                from_token = RelationPaginationToken.from_string(from_token)
+            from_token = None
+            if from_token_str:
+                from_token = RelationPaginationToken.from_string(from_token_str)

-            if to_token:
-                to_token = RelationPaginationToken.from_string(to_token)
+            to_token = None
+            if to_token_str:
+                to_token = RelationPaginationToken.from_string(to_token_str)

            pagination_chunk = await self.store.get_relations_for_event(
                event_id=parent_id,
@@ -256,19 +258,21 @@ class RelationAggregationPaginationServlet(RestServlet):
            raise SynapseError(400, "Relation type must be 'annotation'")

        limit = parse_integer(request, "limit", default=5)
-        from_token = parse_string(request, "from")
-        to_token = parse_string(request, "to")
+        from_token_str = parse_string(request, "from")
+        to_token_str = parse_string(request, "to")

        if event.internal_metadata.is_redacted():
            # If the event is redacted, return an empty list of relations
            pagination_chunk = PaginationChunk(chunk=[])
        else:
            # Return the relations
-            if from_token:
-                from_token = AggregationPaginationToken.from_string(from_token)
+            from_token = None
+            if from_token_str:
+                from_token = AggregationPaginationToken.from_string(from_token_str)

-            if to_token:
-                to_token = AggregationPaginationToken.from_string(to_token)
+            to_token = None
+            if to_token_str:
+                to_token = AggregationPaginationToken.from_string(to_token_str)

            pagination_chunk = await self.store.get_aggregation_groups_for_event(
                event_id=parent_id,
@@ -336,14 +340,16 @@ class RelationAggregationGroupPaginationServlet(RestServlet):
            raise SynapseError(400, "Relation type must be 'annotation'")

        limit = parse_integer(request, "limit", default=5)
-        from_token = parse_string(request, "from")
-        to_token = parse_string(request, "to")
+        from_token_str = parse_string(request, "from")
+        to_token_str = parse_string(request, "to")

-        if from_token:
-            from_token = RelationPaginationToken.from_string(from_token)
+        from_token = None
+        if from_token_str:
+            from_token = RelationPaginationToken.from_string(from_token_str)

-        if to_token:
-            to_token = RelationPaginationToken.from_string(to_token)
+        to_token = None
+        if to_token_str:
+            to_token = RelationPaginationToken.from_string(to_token_str)

        result = await self.store.get_relations_for_event(
            event_id=parent_id,
@@ -112,7 +112,7 @@ class SyncRestServlet(RestServlet):
            default="online",
            allowed_values=self.ALLOWED_PRESENCE,
        )
-        filter_id = parse_string(request, "filter", default=None)
+        filter_id = parse_string(request, "filter")
        full_state = parse_boolean(request, "full_state", default=False)

        logger.debug(
@@ -112,7 +112,7 @@ class ConsentResource(DirectServeHtmlResource):
            request (twisted.web.http.Request):
        """
        version = parse_string(request, "v", default=self._default_consent_version)
-        username = parse_string(request, "u", required=False, default="")
+        username = parse_string(request, "u", default="")
        userhmac = None
        has_consented = False
        public_version = username == ""
@@ -49,6 +49,8 @@ class DownloadResource(DirectServeJsonResource):
            b" media-src 'self';"
            b" object-src 'self';",
        )
+        # Limited non-standard form of CSP for IE11
+        request.setHeader(b"X-Content-Security-Policy", b"sandbox;")
        request.setHeader(
            b"Referrer-Policy",
            b"no-referrer",
@@ -186,15 +186,11 @@ class PreviewUrlResource(DirectServeJsonResource):
        respond_with_json(request, 200, {}, send_cors=True)

    async def _async_render_GET(self, request: SynapseRequest) -> None:
-        # This will always be set by the time Twisted calls us.
-        assert request.args is not None
-
        # XXX: if get_user_by_req fails, what should we do in an async render?
        requester = await self.auth.get_user_by_req(request)
-        url = parse_string(request, "url")
-        if b"ts" in request.args:
-            ts = parse_integer(request, "ts")
-        else:
+        url = parse_string(request, "url", required=True)
+        ts = parse_integer(request, "ts")
+        if ts is None:
            ts = self.clock.time_msec()

        # XXX: we could move this into _do_preview if we wanted.
@@ -16,6 +16,7 @@ import heapq
 import logging
 from collections import defaultdict, namedtuple
 from typing import (
+    TYPE_CHECKING,
    Any,
    Awaitable,
    Callable,
@@ -52,6 +53,10 @@ from synapse.util.async_helpers import Linearizer
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.metrics import Measure, measure_func

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+    from synapse.storage.databases.main import DataStore
+
 logger = logging.getLogger(__name__)
 metrics_logger = logging.getLogger("synapse.state.metrics")

@@ -74,7 +79,7 @@ _NEXT_STATE_ID = 1
 POWER_KEY = (EventTypes.PowerLevels, "")


-def _gen_state_id():
+def _gen_state_id() -> str:
    global _NEXT_STATE_ID
    s = "X%d" % (_NEXT_STATE_ID,)
    _NEXT_STATE_ID += 1
@@ -109,7 +114,7 @@ class _StateCacheEntry:
        # `state_id` is either a state_group (and so an int) or a string. This
        # ensures we don't accidentally persist a state_id as a stateg_group
        if state_group:
-            self.state_id = state_group
+            self.state_id: Union[str, int] = state_group
        else:
            self.state_id = _gen_state_id()

@@ -122,7 +127,7 @@ class StateHandler:
    where necessary
    """

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.clock = hs.get_clock()
        self.store = hs.get_datastore()
        self.state_store = hs.get_storage().state
@@ -507,7 +512,7 @@ class StateResolutionHandler:
    be storage-independent.
    """

-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        self.clock = hs.get_clock()

        self.resolve_linearizer = Linearizer(name="state_resolve_lock")
@@ -636,16 +641,20 @@ class StateResolutionHandler:
        """
        try:
            with Measure(self.clock, "state._resolve_events") as m:
-                v = KNOWN_ROOM_VERSIONS[room_version]
-                if v.state_res == StateResolutionVersions.V1:
+                room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
+                if room_version_obj.state_res == StateResolutionVersions.V1:
                    return await v1.resolve_events_with_store(
-                        room_id, state_sets, event_map, state_res_store.get_events
+                        room_id,
+                        room_version_obj,
+                        state_sets,
+                        event_map,
+                        state_res_store.get_events,
                    )
                else:
                    return await v2.resolve_events_with_store(
                        self.clock,
                        room_id,
-                        room_version,
+                        room_version_obj,
                        state_sets,
                        event_map,
                        state_res_store,
@@ -653,13 +662,15 @@ class StateResolutionHandler:
        finally:
            self._record_state_res_metrics(room_id, m.get_resource_usage())

-    def _record_state_res_metrics(self, room_id: str, rusage: ContextResourceUsage):
+    def _record_state_res_metrics(
+        self, room_id: str, rusage: ContextResourceUsage
+    ) -> None:
        room_metrics = self._state_res_metrics[room_id]
        room_metrics.cpu_time += rusage.ru_utime + rusage.ru_stime
        room_metrics.db_time += rusage.db_txn_duration_sec
        room_metrics.db_events += rusage.evt_db_fetch_count

-    def _report_metrics(self):
+    def _report_metrics(self) -> None:
        if not self._state_res_metrics:
            # no state res has happened since the last iteration: don't bother logging.
            return
@@ -769,16 +780,13 @@ def _make_state_cache_entry(
    )


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class StateResolutionStore:
    """Interface that allows state resolution algorithms to access the database
    in well defined way.
-
-    Args:
-        store (DataStore)
    """

-    store = attr.ib()
+    store: "DataStore"

    def get_events(
        self, event_ids: Iterable[str], allow_rejected: bool = False
@@ -29,7 +29,7 @@ from typing import (
 from synapse import event_auth
 from synapse.api.constants import EventTypes
 from synapse.api.errors import AuthError
-from synapse.api.room_versions import RoomVersions
+from synapse.api.room_versions import RoomVersion, RoomVersions
 from synapse.events import EventBase
 from synapse.types import MutableStateMap, StateMap

@@ -41,6 +41,7 @@ POWER_KEY = (EventTypes.PowerLevels, "")

 async def resolve_events_with_store(
    room_id: str,
+    room_version: RoomVersion,
    state_sets: Sequence[StateMap[str]],
    event_map: Optional[Dict[str, EventBase]],
    state_map_factory: Callable[[Iterable[str]], Awaitable[Dict[str, EventBase]]],
@@ -104,7 +105,7 @@ async def resolve_events_with_store(
    # get the ids of the auth events which allow us to authenticate the
    # conflicted state, picking only from the unconflicting state.
    auth_events = _create_auth_events_from_maps(
-        unconflicted_state, conflicted_state, state_map
+        room_version, unconflicted_state, conflicted_state, state_map
    )

    new_needed_events = set(auth_events.values())
@@ -132,7 +133,7 @@ async def resolve_events_with_store(
    state_map.update(state_map_new)

    return _resolve_with_state(
-        unconflicted_state, conflicted_state, auth_events, state_map
+        room_version, unconflicted_state, conflicted_state, auth_events, state_map
    )


@@ -187,6 +188,7 @@ def _seperate(


 def _create_auth_events_from_maps(
+    room_version: RoomVersion,
    unconflicted_state: StateMap[str],
    conflicted_state: StateMap[Set[str]],
    state_map: Dict[str, EventBase],
@@ -194,6 +196,7 @@ def _create_auth_events_from_maps(
    """

    Args:
+        room_version: The room version.
        unconflicted_state: The unconflicted state map.
        conflicted_state: The conflicted state map.
        state_map:
@@ -205,7 +208,9 @@ def _create_auth_events_from_maps(
    for event_ids in conflicted_state.values():
        for event_id in event_ids:
            if event_id in state_map:
-                keys = event_auth.auth_types_for_event(state_map[event_id])
+                keys = event_auth.auth_types_for_event(
+                    room_version, state_map[event_id]
+                )
                for key in keys:
                    if key not in auth_events:
                        auth_event_id = unconflicted_state.get(key, None)
@@ -215,6 +220,7 @@ def _create_auth_events_from_maps(


 def _resolve_with_state(
+    room_version: RoomVersion,
    unconflicted_state_ids: MutableStateMap[str],
    conflicted_state_ids: StateMap[Set[str]],
    auth_event_ids: StateMap[str],
@@ -235,7 +241,9 @@ def _resolve_with_state(
    }

    try:
-        resolved_state = _resolve_state_events(conflicted_state, auth_events)
+        resolved_state = _resolve_state_events(
+            room_version, conflicted_state, auth_events
+        )
    except Exception:
        logger.exception("Failed to resolve state")
        raise
@@ -248,7 +256,9 @@ def _resolve_with_state(


 def _resolve_state_events(
-    conflicted_state: StateMap[List[EventBase]], auth_events: MutableStateMap[EventBase]
+    room_version: RoomVersion,
+    conflicted_state: StateMap[List[EventBase]],
+    auth_events: MutableStateMap[EventBase],
 ) -> StateMap[EventBase]:
    """This is where we actually decide which of the conflicted state to
    use.
@@ -263,21 +273,27 @@ def _resolve_state_events(
    if POWER_KEY in conflicted_state:
        events = conflicted_state[POWER_KEY]
        logger.debug("Resolving conflicted power levels %r", events)
-        resolved_state[POWER_KEY] = _resolve_auth_events(events, auth_events)
+        resolved_state[POWER_KEY] = _resolve_auth_events(
+            room_version, events, auth_events
+        )

    auth_events.update(resolved_state)

    for key, events in conflicted_state.items():
        if key[0] == EventTypes.JoinRules:
            logger.debug("Resolving conflicted join rules %r", events)
-            resolved_state[key] = _resolve_auth_events(events, auth_events)
+            resolved_state[key] = _resolve_auth_events(
+                room_version, events, auth_events
+            )

    auth_events.update(resolved_state)

    for key, events in conflicted_state.items():
        if key[0] == EventTypes.Member:
            logger.debug("Resolving conflicted member lists %r", events)
-            resolved_state[key] = _resolve_auth_events(events, auth_events)
+            resolved_state[key] = _resolve_auth_events(
+                room_version, events, auth_events
+            )

    auth_events.update(resolved_state)

@@ -290,12 +306,14 @@ def _resolve_state_events(


 def _resolve_auth_events(
-    events: List[EventBase], auth_events: StateMap[EventBase]
+    room_version: RoomVersion, events: List[EventBase], auth_events: StateMap[EventBase]
 ) -> EventBase:
    reverse = list(reversed(_ordered_events(events)))

    auth_keys = {
-        key for event in events for key in event_auth.auth_types_for_event(event)
+        key
+        for event in events
+        for key in event_auth.auth_types_for_event(room_version, event)
    }

    new_auth_events = {}
@@ -36,7 +36,7 @@ import synapse.state
 from synapse import event_auth
 from synapse.api.constants import EventTypes
 from synapse.api.errors import AuthError
-from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
+from synapse.api.room_versions import RoomVersion
 from synapse.events import EventBase
 from synapse.types import MutableStateMap, StateMap
 from synapse.util import Clock
@@ -53,7 +53,7 @@ _AWAIT_AFTER_ITERATIONS = 100
 async def resolve_events_with_store(
    clock: Clock,
    room_id: str,
-    room_version: str,
+    room_version: RoomVersion,
    state_sets: Sequence[StateMap[str]],
    event_map: Optional[Dict[str, EventBase]],
    state_res_store: "synapse.state.StateResolutionStore",
@@ -497,7 +497,7 @@ async def _reverse_topological_power_sort(
 async def _iterative_auth_checks(
    clock: Clock,
    room_id: str,
-    room_version: str,
+    room_version: RoomVersion,
    event_ids: List[str],
    base_state: StateMap[str],
    event_map: Dict[str, EventBase],
@@ -519,7 +519,6 @@ async def _iterative_auth_checks(
        Returns the final updated state
    """
    resolved_state = dict(base_state)
-    room_version_obj = KNOWN_ROOM_VERSIONS[room_version]

    for idx, event_id in enumerate(event_ids, start=1):
        event = event_map[event_id]
@@ -538,7 +537,7 @@ async def _iterative_auth_checks(
                if ev.rejected_reason is None:
                    auth_events[(ev.type, ev.state_key)] = ev

-        for key in event_auth.auth_types_for_event(event):
+        for key in event_auth.auth_types_for_event(room_version, event):
            if key in resolved_state:
                ev_id = resolved_state[key]
                ev = await _get_event(room_id, ev_id, event_map, state_res_store)
@@ -548,7 +547,7 @@ async def _iterative_auth_checks(

        try:
            event_auth.check(
-                room_version_obj,
+                room_version,
                event,
                auth_events,
                do_sig_check=False,
@@ -832,31 +832,16 @@ class DatabasePool:
        self,
        table: str,
        values: Dict[str, Any],
-        or_ignore: bool = False,
        desc: str = "simple_insert",
-    ) -> bool:
+    ) -> None:
        """Executes an INSERT query on the named table.

        Args:
            table: string giving the table name
            values: dict of new column names and values for them
-            or_ignore: bool stating whether an exception should be raised
-                when a conflicting row already exists. If True, False will be
-                returned by the function instead
            desc: description of the transaction, for logging and metrics
-
-        Returns:
-             Whether the row was inserted or not. Only useful when `or_ignore` is True
        """
-        try:
-            await self.runInteraction(desc, self.simple_insert_txn, table, values)
-        except self.engine.module.IntegrityError:
-            # We have to do or_ignore flag at this layer, since we can't reuse
-            # a cursor after we receive an error from the db.
-            if not or_ignore:
-                raise
-            return False
-        return True
+        await self.runInteraction(desc, self.simple_insert_txn, table, values)

    @staticmethod
    def simple_insert_txn(
@@ -930,7 +915,7 @@ class DatabasePool:
        insertion_values: Optional[Dict[str, Any]] = None,
        desc: str = "simple_upsert",
        lock: bool = True,
-    ) -> Optional[bool]:
+    ) -> bool:
        """

        `lock` should generally be set to True (the default), but can be set
@@ -951,8 +936,8 @@ class DatabasePool:
            desc: description of the transaction, for logging and metrics
            lock: True to lock the table when doing the upsert.
        Returns:
-            Native upserts always return None. Emulated upserts return True if a
-            new entry was created, False if an existing one was updated.
+            Returns True if a row was inserted or updated (i.e. if `values` is
+            not empty then this always returns True)
        """
        insertion_values = insertion_values or {}

@@ -995,7 +980,7 @@ class DatabasePool:
        values: Dict[str, Any],
        insertion_values: Optional[Dict[str, Any]] = None,
        lock: bool = True,
-    ) -> Optional[bool]:
+    ) -> bool:
        """
        Pick the UPSERT method which works best on the platform. Either the
        native one (Pg9.5+, recent SQLites), or fall back to an emulated method.
@@ -1008,16 +993,15 @@ class DatabasePool:
            insertion_values: additional key/values to use only when inserting
            lock: True to lock the table when doing the upsert.
        Returns:
-            Native upserts always return None. Emulated upserts return True if a
-            new entry was created, False if an existing one was updated.
+            Returns True if a row was inserted or updated (i.e. if `values` is
+            not empty then this always returns True)
        """
        insertion_values = insertion_values or {}

        if self.engine.can_native_upsert and table not in self._unsafe_to_upsert_tables:
-            self.simple_upsert_txn_native_upsert(
+            return self.simple_upsert_txn_native_upsert(
                txn, table, keyvalues, values, insertion_values=insertion_values
            )
-            return None
        else:
            return self.simple_upsert_txn_emulated(
                txn,
@@ -1045,8 +1029,8 @@ class DatabasePool:
            insertion_values: additional key/values to use only when inserting
            lock: True to lock the table when doing the upsert.
        Returns:
-            Returns True if a new entry was created, False if an existing
-            one was updated.
+            Returns True if a row was inserted or updated (i.e. if `values` is
+            not empty then this always returns True)
        """
        insertion_values = insertion_values or {}

@@ -1086,8 +1070,7 @@ class DatabasePool:

            txn.execute(sql, sqlargs)
            if txn.rowcount > 0:
-                # successfully updated at least one row.
-                return False
+                return True

        # We didn't find any existing rows, so insert a new one
        allvalues: Dict[str, Any] = {}
@@ -1111,15 +1094,19 @@ class DatabasePool:
        keyvalues: Dict[str, Any],
        values: Dict[str, Any],
        insertion_values: Optional[Dict[str, Any]] = None,
-    ) -> None:
+    ) -> bool:
        """
-        Use the native UPSERT functionality in recent PostgreSQL versions.
+        Use the native UPSERT functionality in PostgreSQL.

        Args:
            table: The table to upsert into
            keyvalues: The unique key tables and their new values
            values: The nonunique columns and their new values
            insertion_values: additional key/values to use only when inserting
+
+        Returns:
+            Returns True if a row was inserted or updated (i.e. if `values` is
+            not empty then this always returns True)
        """
        allvalues: Dict[str, Any] = {}
        allvalues.update(keyvalues)
@@ -1140,6 +1127,8 @@ class DatabasePool:
        )
        txn.execute(sql, list(allvalues.values()))

+        return bool(txn.rowcount)
+
    async def simple_upsert_many(
        self,
        table: str,
@@ -249,7 +249,7 @@ class DataStore(
        name: Optional[str] = None,
        guests: bool = True,
        deactivated: bool = False,
-        order_by: UserSortOrder = UserSortOrder.USER_ID.value,
+        order_by: str = UserSortOrder.USER_ID.value,
        direction: str = "f",
    ) -> Tuple[List[JsonDict], int]:
        """Function to retrieve a paginated list of users from
@@ -297,27 +297,22 @@ class DataStore(

            where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""

-            sql_base = """
+            sql_base = f"""
                FROM users as u
                LEFT JOIN profiles AS p ON u.name = '@' || p.user_id || ':' || ?
-                {}
-                """.format(
-                where_clause
-            )
+                {where_clause}
+                """
            sql = "SELECT COUNT(*) as total_users " + sql_base
            txn.execute(sql, args)
            count = txn.fetchone()[0]

-            sql = """
-                SELECT name, user_type, is_guest, admin, deactivated, shadow_banned, displayname, avatar_url
+            sql = f"""
+                SELECT name, user_type, is_guest, admin, deactivated, shadow_banned,
+                displayname, avatar_url, creation_ts * 1000 as creation_ts
                {sql_base}
                ORDER BY {order_by_column} {order}, u.name ASC
                LIMIT ? OFFSET ?
-            """.format(
-                sql_base=sql_base,
-                order_by_column=order_by_column,
-                order=order,
-            )
+            """
            args += [limit, start]
            txn.execute(sql, args)
            users = self.db_pool.cursor_to_dict(txn)
@@ -1078,16 +1078,18 @@ class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
            return False

        try:
-            inserted = await self.db_pool.simple_insert(
+            inserted = await self.db_pool.simple_upsert(
                "devices",
-                values={
+                keyvalues={
                    "user_id": user_id,
                    "device_id": device_id,
+                },
+                values={},
+                insertion_values={
                    "display_name": initial_device_display_name,
                    "hidden": False,
                },
                desc="store_device",
-                or_ignore=True,
            )
            if not inserted:
                # if the device already exists, check if it's a real device, or
@@ -1099,6 +1101,7 @@ class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
                )
                if hidden:
                    raise StoreError(400, "The device ID is in use", Codes.FORBIDDEN)
+
            self.device_id_exists_cache.set(key, True)
            return inserted
        except StoreError:
@@ -21,7 +21,6 @@ from canonicaljson import encode_canonical_json

 from twisted.enterprise.adbapi import Connection

-from synapse.api.constants import DeviceKeyAlgorithms
 from synapse.logging.opentracing import log_kv, set_tag, trace
 from synapse.storage._base import SQLBaseStore, db_to_json
 from synapse.storage.database import DatabasePool, make_in_list_sql_clause
@@ -382,15 +381,9 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore):
                " GROUP BY algorithm"
            )
            txn.execute(sql, (user_id, device_id))
-
-            # Initially set the key count to 0. This ensures that the client will always
-            # receive *some count*, even if it's 0.
-            result = {DeviceKeyAlgorithms.SIGNED_CURVE25519: 0}
-
-            # Override entries with the count of any keys we pulled from the database
+            result = {}
            for algorithm, key_count in txn:
                result[algorithm] = key_count
-
            return result

        return await self.db_pool.runInteraction(
@@ -297,17 +297,13 @@ class MonthlyActiveUsersStore(MonthlyActiveUsersWorkerStore):
        Args:
            txn (cursor):
            user_id (str): user to add/update
-
-        Returns:
-            bool: True if a new entry was created, False if an
-            existing one was updated.
        """

        # Am consciously deciding to lock the table on the basis that is ought
        # never be a big table and alternative approaches (batching multiple
        # upserts into a single txn) introduced a lot of extra complexity.
        # See https://github.com/matrix-org/synapse/issues/3854 for more
-        is_insert = self.db_pool.simple_upsert_txn(
+        self.db_pool.simple_upsert_txn(
            txn,
            table="monthly_active_users",
            keyvalues={"user_id": user_id},
@@ -322,8 +318,6 @@ class MonthlyActiveUsersStore(MonthlyActiveUsersWorkerStore):
            txn, self.user_last_seen_monthly_active, (user_id,)
        )

-        return is_insert
-
    async def populate_monthly_active_users(self, user_id):
        """Checks on the state of monthly active user limits and optionally
        add the user to the monthly active tables
@@ -363,7 +363,7 @@ class RoomWorkerStore(SQLBaseStore):
        self,
        start: int,
        limit: int,
-        order_by: RoomSortOrder,
+        order_by: str,
        reverse_order: bool,
        search_term: Optional[str],
    ) -> Tuple[List[Dict[str, Any]], int]:
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Erik Johnston	a48296dd86	WIP docs	2021-07-28 11:06:24 +01:00
Erik Johnston	13f9422e38	Allow /typing to be handled by any worker	2021-07-28 10:58:45 +01:00
reivilibre	e16eab29d6	Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407 ) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-27 14:32:05 +01:00
Patrick Cloke	13944678c3	Use new go test running syntax for complement. (#10488 ) Updates CI and the helper script t ensures all tests are run (in parallel).	2021-07-27 12:08:51 +00:00
Denis Kasak	2476d5373c	Mitigate media repo XSSs on IE11. (#10468 ) IE11 doesn't support Content-Security-Policy but it has support for a non-standard X-Content-Security-Policy header, which only supports the sandbox directive. This prevents script execution, so it at least offers some protection against media repo-based attacks. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>	2021-07-27 13:45:10 +02:00
Travis Ralston	b3a757eb3b	Support MSC2033: Device ID on whoami (#9918 ) * Fix no-access-token bug in deactivation tests * Support MSC2033: Device ID on whoami * Test for appservices too MSC: https://github.com/matrix-org/matrix-doc/pull/2033 The MSC has passed FCP, which means stable endpoints can be used.	2021-07-27 05:28:20 +00:00
Patrick Cloke	b7186c6e8d	Add type hints to state handler. (#10482 )	2021-07-26 12:49:53 -04:00
Patrick Cloke	228decfce1	Update the MSC3083 support to verify if joins are from an authorized server. (#10254 )	2021-07-26 12:17:00 -04:00
Patrick Cloke	4fb92d93ea	Add type hints to synapse.federation.transport.client. (#10408 )	2021-07-26 11:53:09 -04:00
Richard van der Hoff	f22252d4f9	Enable docker image caching for the deb build (#10431 )	2021-07-26 11:36:01 +01:00
Erik Johnston	ab82fd6ed1	Merge branch 'release-v1.39' into develop	2021-07-23 09:19:24 +01:00
Erik Johnston	c39a417de0	Merge tag 'v1.39.0rc2' into develop Synapse 1.39.0rc2 (2021-07-22) ============================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457)) Internal Changes ---------------- - Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))	2021-07-23 09:04:41 +01:00
Erik Johnston	683deee9a4	Merge branch 'master' into develop	2021-07-23 09:03:19 +01:00
Richard van der Hoff	016f085722	Merge tag 'v1.38.1' Synapse 1.38.1 (2021-07-22) =========================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))	2021-07-23 00:43:53 +01:00
Eric Eastwood	cd5fcd2731	Disable msc2716 until Complement update is merged (#10463 )	2021-07-22 20:19:30 +00:00
Dirk Klimpel	89c4ca81bb	Add `creation_ts` to list users admin API (#10448 ) Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-07-22 16:05:16 +02:00
Erik Johnston	38b346a504	Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442 ) Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.	2021-07-22 12:39:50 +01:00
Richard van der Hoff	d8324b8238	Fix a handful of type annotations. (#10446 ) * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog	2021-07-22 12:00:16 +01:00
Eric Eastwood	d518b05a86	Move dev/ docs to development/ (#10453 )	2021-07-22 12:58:24 +02:00
Richard van der Hoff	5e2df47f72	Cancel redundant GHA workflows (#10451 )	2021-07-22 11:35:06 +01:00
Richard van der Hoff	f1347bcfdc	Fix the tests-done Github Actions job (#10444 )	2021-07-22 11:10:30 +01:00
Richard van der Hoff	8ae0bdca75	Drop xenial-support hacks (#10429 )	2021-07-21 21:25:28 +01:00
Patrick Cloke	590cc4e888	Add type hints to additional servlet functions (#10437 ) Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.	2021-07-21 18:12:22 +00:00
Patrick Cloke	5b68816de9	Fix the hierarchy of OpenID providers in the docs. (#10445 )	2021-07-21 13:48:06 -04:00
Patrick Cloke	d15e72e511	Update the notification email subject when invited to a space. (#10426 )	2021-07-21 17:29:54 +00:00
Richard van der Hoff	b2629e7016	Merge remote-tracking branch 'origin/release-v1.39' into develop	2021-07-21 16:12:23 +01:00
Patrick Cloke	5db118626b	Add a return type to parse_string. (#10438 ) And set the required attribute in a few places which will error if a parameter is not provided.	2021-07-21 09:47:56 -04:00
Eric Eastwood	2d89c66b88	Switch to `chunk` events so we can auth via power_levels (MSC2716) (#10432 ) Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room.	2021-07-21 10:29:57 +00:00
Andrew Morgan	b181dc402d	Merge tag 'v1.39.0rc1' into develop Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))	2021-07-20 16:47:44 +01:00
Michael Telatynski	69226c1ab4	MSC3244 room capabilities implementation (#10283 )	2021-07-20 12:59:23 +01:00
Erik Johnston	794371b1bf	Revert "Fix dropping locks on shut down" This reverts commit `83f1ccfcab`.	2021-07-20 12:28:40 +01:00
Erik Johnston	83f1ccfcab	Fix dropping locks on shut down	2021-07-20 12:28:00 +01:00
				`@@ -0,0 +1 @@`
				`Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events.`
				`@@ -0,0 +1 @@`
				`Initial support for MSC3244, Room version capabilities over the /capabilities API.`
				`@@ -0,0 +1 @@`
				`Add a buffered logging handler which periodically flushes itself.`
				`@@ -0,0 +1 @@`
				Add type hints to `synapse.federation.transport.client` module.
				`@@ -0,0 +1 @@`
				`Email notifications now state whether an invitation is to a room or a space.`
				`@@ -0,0 +1 @@`
				`Drop backwards-compatibility code that was required to support Ubuntu Xenial.`
				`@@ -0,0 +1 @@`
				`Use a docker image cache for the prerequisites for the debian package build.`
				`@@ -0,0 +1 @@`
				`Connect historical chunks together with chunk events instead of a content field (MSC2716).`
				`@@ -0,0 +1 @@`
				Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages.