Merge branch 'develop' into madlittlemods/event_id_always_failed_to_fetch

Conflicts: synapse/handlers/federation_event.py
Keep track when we try and fail to process a pulled event (#13589 )
2022-09-14 18:14:17 -05:00 · 2022-09-14 13:57:50 -05:00 · 2022-09-14 17:11:16 +00:00 · 2022-09-14 12:01:42 -04:00 · 2022-09-14 15:53:18 +00:00 · 2022-09-14 15:29:05 +00:00
129 changed files with 1419 additions and 645 deletions
@@ -468,10 +468,6 @@ jobs:
  tests-done:
    if: ${{ always() }}
    needs:
-      - check-sampleconfig
-      - lint
-      - lint-crlf
-      - lint-newsfile
      - trial
      - trial-olddeps
      - sytest
@@ -486,5 +482,7 @@ jobs:
          needs: ${{ toJSON(needs) }}

          # The newsfile lint may be skipped on non PR builds
-          skippable:
+          # Cargo test is skipped if there is no changes on Rust code
+          skippable: |
            lint-newsfile
+            cargo-test
@@ -1,3 +1,111 @@
+Synapse 1.67.0 (2022-09-13)
+===========================
+
+This release removes using the deprecated direct TCP replication configuration
+for workers. Server admins should use Redis instead. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+The minimum version of `poetry` supported for managing source checkouts is now
+1.2.0.
+
+**Notice:** from the next major release (1.68.0) installing Synapse from a source
+checkout will require a recent Rust compiler. Those using packages or
+`pip install matrix-synapse` will not be affected. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+**Notice:** from the next major release (1.68.0), running Synapse with a SQLite
+database will require SQLite version 3.27.0 or higher. (The [current minimum
+ version is SQLite 3.22.0](https://github.com/matrix-org/synapse/blob/release-v1.67/synapse/storage/engines/sqlite.py#L69-L78).)
+See [#12983](https://github.com/matrix-org/synapse/issues/12983) and the [upgrade notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670) for more details.
+
+
+No significant changes since 1.67.0rc1.
+
+
+Synapse 1.67.0rc1 (2022-09-06)
+==============================
+
+Features
+--------
+
+- Support setting the registration shared secret in a file, via a new `registration_shared_secret_path` configuration option. ([\#13614](https://github.com/matrix-org/synapse/issues/13614))
+- Change the default startup behaviour so that any missing "additional" configuration files (signing key, etc) are generated automatically. ([\#13615](https://github.com/matrix-org/synapse/issues/13615))
+- Improve performance of sending messages in rooms with thousands of local users. ([\#13634](https://github.com/matrix-org/synapse/issues/13634))
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database. ([\#13509](https://github.com/matrix-org/synapse/issues/13509))
+- Fix bug that user cannot `/forget` rooms after the last member has left the room. ([\#13546](https://github.com/matrix-org/synapse/issues/13546))
+- Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room. ([\#13583](https://github.com/matrix-org/synapse/issues/13583))
+- Fix loading the current stream position behind the actual position. ([\#13585](https://github.com/matrix-org/synapse/issues/13585))
+- Fix a longstanding bug in `register_new_matrix_user` which meant it was always necessary to explicitly give a server URL. ([\#13616](https://github.com/matrix-org/synapse/issues/13616))
+- Fix the running of [MSC1763](https://github.com/matrix-org/matrix-spec-proposals/pull/1763) retention purge_jobs in deployments with background jobs running on a worker by forcing them back onto the main worker. Contributed by Brad @ Beeper. ([\#13632](https://github.com/matrix-org/synapse/issues/13632))
+- Fix a long-standing bug that downloaded media for URL previews was not deleted while database background updates were running. ([\#13657](https://github.com/matrix-org/synapse/issues/13657))
+- Fix [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint to return the correct next event when the events have the same timestamp. ([\#13658](https://github.com/matrix-org/synapse/issues/13658))
+- Fix bug where we wedge media plugins if clients disconnect early. Introduced in v1.22.0. ([\#13660](https://github.com/matrix-org/synapse/issues/13660))
+- Fix a long-standing bug which meant that keys for unwhitelisted servers were not returned by `/_matrix/key/v2/query`. ([\#13683](https://github.com/matrix-org/synapse/issues/13683))
+- Fix a bug introduced in Synapse v1.20.0 that would cause the unstable unread counts from [MSC2654](https://github.com/matrix-org/matrix-spec-proposals/pull/2654) to be calculated even if the feature is disabled. ([\#13694](https://github.com/matrix-org/synapse/issues/13694))
+
+
+Updates to the Docker image
+---------------------------
+
+- Update docker image to use a stable version of poetry. ([\#13688](https://github.com/matrix-org/synapse/issues/13688))
+
+
+Improved Documentation
+----------------------
+
+- Improve the description of the ["chain cover index"](https://matrix-org.github.io/synapse/latest/auth_chain_difference_algorithm.html) used internally by Synapse. ([\#13602](https://github.com/matrix-org/synapse/issues/13602))
+- Document how ["monthly active users"](https://matrix-org.github.io/synapse/latest/usage/administration/monthly_active_users.html) is calculated and used. ([\#13617](https://github.com/matrix-org/synapse/issues/13617))
+- Improve documentation around user registration. ([\#13640](https://github.com/matrix-org/synapse/issues/13640))
+- Remove documentation of legacy `frontend_proxy` worker app. ([\#13645](https://github.com/matrix-org/synapse/issues/13645))
+- Clarify documentation that HTTP replication traffic can be protected with a shared secret. ([\#13656](https://github.com/matrix-org/synapse/issues/13656))
+- Remove unintentional colons from [config manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html) headers. ([\#13665](https://github.com/matrix-org/synapse/issues/13665))
+- Update docs to make enabling metrics more clear. ([\#13678](https://github.com/matrix-org/synapse/issues/13678))
+- Clarify `(room_id, event_id)` global uniqueness and how we should scope our database schemas. ([\#13701](https://github.com/matrix-org/synapse/issues/13701))
+
+
+Deprecations and Removals
+-------------------------
+
+- Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\#13241](https://github.com/matrix-org/synapse/issues/13241))
+- Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar). ([\#13569](https://github.com/matrix-org/synapse/issues/13569))
+- Remove the ability to use direct TCP replication with workers. Direct TCP replication was deprecated in Synapse v1.18.0. Workers now require using Redis. ([\#13647](https://github.com/matrix-org/synapse/issues/13647))
+- Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). ([\#13653](https://github.com/matrix-org/synapse/issues/13653), [\#13692](https://github.com/matrix-org/synapse/issues/13692))
+
+
+Internal Changes
+----------------
+
+- Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process. ([\#13483](https://github.com/matrix-org/synapse/issues/13483))
+- Add experimental configuration option to allow disabling legacy Prometheus metric names. ([\#13540](https://github.com/matrix-org/synapse/issues/13540))
+- Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar). ([\#13573](https://github.com/matrix-org/synapse/issues/13573), [\#13600](https://github.com/matrix-org/synapse/issues/13600))
+- Optimize how Synapse calculates domains to fetch from during backfill. ([\#13575](https://github.com/matrix-org/synapse/issues/13575))
+- Comment about a better future where we can get the state diff between two events. ([\#13586](https://github.com/matrix-org/synapse/issues/13586))
+- Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls for understandable traces in Jaeger. ([\#13588](https://github.com/matrix-org/synapse/issues/13588))
+- Improve performance of `@cachedList`. ([\#13591](https://github.com/matrix-org/synapse/issues/13591))
+- Minor speed up of fetching large numbers of push rules. ([\#13592](https://github.com/matrix-org/synapse/issues/13592))
+- Optimise push action fetching queries. Contributed by Nick @ Beeper (@fizzadar). ([\#13597](https://github.com/matrix-org/synapse/issues/13597))
+- Rename `event_map` to `unpersisted_events` when computing the auth differences. ([\#13603](https://github.com/matrix-org/synapse/issues/13603))
+- Refactor `get_users_in_room(room_id)` mis-use with dedicated `get_current_hosts_in_room(room_id)` function. ([\#13605](https://github.com/matrix-org/synapse/issues/13605))
+- Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request. ([\#13606](https://github.com/matrix-org/synapse/issues/13606))
+- Refactor `get_users_in_room(room_id)` mis-use to lookup single local user with dedicated `check_local_user_in_room(...)` function. ([\#13608](https://github.com/matrix-org/synapse/issues/13608))
+- Drop unused column `application_services_state.last_txn`. ([\#13627](https://github.com/matrix-org/synapse/issues/13627))
+- Improve readability of Complement CI logs by printing failure results last. ([\#13639](https://github.com/matrix-org/synapse/issues/13639))
+- Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods. ([\#13662](https://github.com/matrix-org/synapse/issues/13662))
+- Introduce a `CommonUsageMetrics` class to share some usage metrics between the Prometheus exporter and the phone home stats. ([\#13671](https://github.com/matrix-org/synapse/issues/13671))
+- Add some logging to help track down #13444. ([\#13679](https://github.com/matrix-org/synapse/issues/13679))
+- Update poetry lock file for v1.2.0. ([\#13689](https://github.com/matrix-org/synapse/issues/13689))
+- Add cache to `is_partial_state_room`. ([\#13693](https://github.com/matrix-org/synapse/issues/13693))
+- Update the Grafana dashboard that is included with Synapse in the `contrib` directory. ([\#13697](https://github.com/matrix-org/synapse/issues/13697))
+- Only run trial CI on all python versions on non-PRs. ([\#13698](https://github.com/matrix-org/synapse/issues/13698))
+- Fix typechecking with latest types-jsonschema. ([\#13712](https://github.com/matrix-org/synapse/issues/13712))
+- Reduce number of CI checks we run for PRs. ([\#13713](https://github.com/matrix-org/synapse/issues/13713))
+
+
 Synapse 1.66.0 (2022-08-31)
 ===========================

@@ -3,7 +3,3 @@

 [workspace]
 members = ["rust"]
-
-
-[profile.release]
-debug = true
@@ -1 +0,0 @@
-Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
@@ -0,0 +1 @@
+Prevent `Failed to fetch missing state events...` warning being logged every time we tried to fetch `state_ids` for a missing `prev_event`.
@@ -0,0 +1 @@
+Note that `libpq` is required on ARM-based Macs.
@@ -1 +0,0 @@
-Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process.
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database.
@@ -1 +0,0 @@
-Add experimental configuration option to allow disabling legacy Prometheus metric names.
@@ -1 +0,0 @@
-Fix bug that user cannot `/forget` rooms after the last member has left the room.
@@ -1 +0,0 @@
-Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Optimize how Synapse calculates domains to fetch from during backfill.
@@ -1 +0,0 @@
-Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room.
@@ -1 +0,0 @@
-Fix loading the current stream position behind the actual position.
@@ -1 +0,0 @@
-Comment about a better future where we can get the state diff between two events.
@@ -1 +0,0 @@
-Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls for understandable traces in Jaeger.
@@ -0,0 +1 @@
+Keep track when we attempt to backfill an event but fail so we can intelligently back-off in the future.
@@ -1 +0,0 @@
-Improve performance of `@cachedList`.
@@ -1 +0,0 @@
-Minor speed up of fetching large numbers of push rules.
@@ -1 +0,0 @@
- Optimise push action fetching queries. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Improve the description of the ["chain cover index"](https://matrix-org.github.io/synapse/latest/auth_chain_difference_algorithm.html) used internally by Synapse.
@@ -1 +0,0 @@
-Rename `event_map` to `unpersisted_events` when computing the auth differences.
@@ -1 +0,0 @@
-Refactor `get_users_in_room(room_id)` mis-use with dedicated `get_current_hosts_in_room(room_id)` function.
@@ -1 +0,0 @@
-Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request.
@@ -1 +0,0 @@
-Refactor `get_users_in_room(room_id)` mis-use to lookup single local user with dedicated `check_local_user_in_room(...)` function.
@@ -1 +0,0 @@
-Support setting the registration shared secret in a file, via a new `registration_shared_secret_path` configuration option.
@@ -1 +0,0 @@
-Change the default startup behaviour so that any missing "additional" configuration files (signing key, etc) are generated automatically.
@@ -1 +0,0 @@
-Fix a longstanding bug in `register_new_matrix_user` which meant it was always necessary to explicitly give a server URL.
@@ -1 +0,0 @@
-Document how ["monthly active users"](https://matrix-org.github.io/synapse/latest/usage/administration/monthly_active_users.html) is calculated and used.
@@ -1 +0,0 @@
-Drop unused column `application_services_state.last_txn`.
@@ -1 +0,0 @@
-Fix the running of MSC1763 retention purge_jobs in deployments with background jobs running on a worker by forcing them back onto the main worker. Contributed by Brad @ Beeper.
@@ -1 +0,0 @@
-Improve performance of sending messages in rooms with thousands of local users.
@@ -1 +0,0 @@
-Improve readability of Complement CI logs by printing failure results last.
@@ -1 +0,0 @@
-Improve documentation around user registration.
@@ -1 +0,0 @@
-Remove documentation of legacy `frontend_proxy` worker app.
@@ -1 +0,0 @@
-Remove the ability to use direct TCP replication with workers. Direct TCP replication was deprecated in Synapse v1.18.0. Workers now require using Redis.
@@ -1 +0,0 @@
-Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285).
@@ -1 +0,0 @@
-Clarify documentation that HTTP replication traffic can be protected with a shared secret.
@@ -1 +0,0 @@
-Fix a long-standing bug that downloaded media for URL previews was not deleted while database background updates were running.
@@ -1 +0,0 @@
-Fix MSC3030 `/timestamp_to_event` endpoint to return the correct next event when the events have the same timestamp.
@@ -1 +0,0 @@
-Fix bug where we wedge media plugins if clients disconnect early. Introduced in v1.22.0.
@@ -1 +0,0 @@
-Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods.
@@ -1 +0,0 @@
-Remove unintentional colons from [config manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html) headers.
@@ -1 +0,0 @@
-Introduce a `CommonUsageMetrics` class to share some usage metrics between the Prometheus exporter and the phone home stats.
@@ -1 +0,0 @@
-Update docs to make enabling metrics more clear.
@@ -1 +0,0 @@
-Add some logging to help track down #13444.
@@ -1 +0,0 @@
-Fix a long-standing bug which meant that keys for unwhitelisted servers were not returned by `/_matrix/key/v2/query`.
@@ -1 +0,0 @@
-Update docker image to use a stable version of poetry.
@@ -1 +0,0 @@
-Update poetry lock file for v1.2.0.
@@ -1 +0,0 @@
-Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285).
@@ -1 +0,0 @@
-Add cache to `is_partial_state_room`.
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse v1.20.0 that would cause the unstable unread counts from [MSC2654](https://github.com/matrix-org/matrix-spec-proposals/pull/2654) to be calculated even if the feature is disabled.
@@ -1 +0,0 @@
-Update the Grafana dashboard that is included with Synapse in the `contrib` directory.
@@ -1 +0,0 @@
-Only run trial CI on all python versions on non-PRs.
@@ -1 +0,0 @@
-Clarify `(room_id, event_id)` global uniqueness and how we should scope our database schemas.
@@ -0,0 +1 @@
+Add & populate `event_stream_ordering` column on receipts table for future optimisation of push action processing. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Fix typechecking with latest types-jsonschema.
@@ -1 +0,0 @@
-Reduce number of CI checks we run for PRs.
@@ -0,0 +1 @@
+Fix a typo in the documentation for the login ratelimiting configuration.
@@ -0,0 +1 @@
+Strip number suffix from instance name to consolidate services that traces are spread over.
@@ -1 +0,0 @@
-Convert `LruCache` linked lists into Rust.
@@ -0,0 +1 @@
+Remove old queries to join room memberships to current state events. Contributed by Nick @ Beeper (@fizzadar).
@@ -0,0 +1 @@
+Fix a long standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver.
@@ -0,0 +1 @@
+Prepatory work for storing thread IDs for notifications and receipts.
@@ -0,0 +1 @@
+Add a check for editable installs if the Rust library needs rebuilding.
@@ -0,0 +1 @@
+Tag traces with the instance name to be able to easily jump into the right logs and filter traces by instance.
@@ -0,0 +1 @@
+Concurrently fetch room push actions when calculating badge counts. Contributed by Nick @ Beeper (@fizzadar).
@@ -0,0 +1 @@
+Fix a long-standing bug where the `cache_invalidation_stream_seq` sequence would begin at 1 instead of 2.
@@ -0,0 +1 @@
+Add a stub Rust crate.
@@ -0,0 +1 @@
+Update the script which makes full schema dumps.
@@ -0,0 +1 @@
+Add a stub Rust crate.
@@ -0,0 +1 @@
+Deduplicate `is_server_notices_room`.
@@ -0,0 +1 @@
+Simplify the dependency DAG in the tests workflow.
@@ -0,0 +1 @@
+Remove an old, incorrect migration file.
@@ -0,0 +1 @@
+Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests.
@@ -0,0 +1 @@
+Remove unused method in `synapse.api.auth.Auth`.
@@ -0,0 +1 @@
+Fix a memory leak when running the unit tests.
@@ -0,0 +1 @@
+Use partial indices on SQLite.
@@ -1,8 +1,18 @@
-matrix-synapse-py3 (1.66.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.67.0) stable; urgency=medium

+  * New Synapse release 1.67.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 13 Sep 2022 09:19:56 +0100
+
+matrix-synapse-py3 (1.67.0~rc1) stable; urgency=medium
+
+  [ Erik Johnston ]
  * Use stable poetry 1.2.0 version, rather than a prerelease.

- -- Erik Johnston <erik@matrix.org>  Thu, 01 Sep 2022 13:48:31 +0100
+  [ Synapse Packaging team ]
+  * New Synapse release 1.67.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Sep 2022 09:01:06 +0100

 matrix-synapse-py3 (1.66.0) stable; urgency=medium

@@ -303,9 +303,10 @@ You may need to install the latest Xcode developer tools:
 xcode-select --install
 ```

-On ARM-based Macs you may need to explicitly install libjpeg which is a pillow dependency. You can use Homebrew (https://brew.sh):
+On ARM-based Macs you may need to install libjpeg and libpq. 
+You can use Homebrew (https://brew.sh):
 ```sh
- brew install jpeg
+ brew install jpeg libpq
 ```

 On macOS Catalina (10.15) you may need to explicitly install OpenSSL
@@ -111,6 +111,30 @@ and remove the TCP `replication` listener from config of the master and
 The minimum supported version of poetry is now 1.2. This should only affect
 those installing from a source checkout.

+## Rust requirement in the next release
+
+From the next major release (v1.68.0) installing Synapse from a source checkout
+will require a recent Rust compiler. Those using packages or
+`pip install matrix-synapse` will not be affected.
+
+The simplest way of installing Rust is via [rustup.rs](https://rustup.rs/)
+
+## SQLite version requirement in the next release
+
+From the next major release (v1.68.0) Synapse will require SQLite 3.27.0 or 
+higher. Synapse v1.67.0 will be the last major release supporting SQLite
+versions 3.22 to 3.26.
+
+Those using docker images or Debian packages from Matrix.org will not be
+affected. If you have installed from source, you should check the version of 
+SQLite used by Python with:
+
+```shell
+python -c "import sqlite3; print(sqlite3.sqlite_version)"
+```
+
+If this is too old, refer to your distribution for advice on upgrading.
+
 # Upgrading to v1.66.0

 ## Delegation of email validation no longer supported
@@ -1393,7 +1393,7 @@ This option specifies several limits for login:
  client is attempting to log into. Defaults to `per_second: 0.17`,
  `burst_count: 3`.

-* `failted_attempts` ratelimits login requests based on the account the
+* `failed_attempts` ratelimits login requests based on the account the
  client is attempting to log into, based on the amount of failed login
  attempts for this account. Defaults to `per_second: 0.17`, `burst_count: 3`.

@@ -57,7 +57,7 @@ manifest-path = "rust/Cargo.toml"

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.66.0"
+version = "1.67.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -18,7 +18,8 @@ crate-type = ["cdylib"]
 name = "synapse.synapse_rust"

 [dependencies]
-intrusive-collections = "0.9.4"
-lazy_static = "1.4.0"
-log = "0.4.17"
 pyo3 = { version = "0.16.5", features = ["extension-module", "macros", "abi3", "abi3-py37"] }
+
+[build-dependencies]
+blake2 = "0.10.4"
+hex = "0.4.3"
@@ -0,0 +1,45 @@
+//! This build script calculates the hash of all files in the `src/`
+//! directory and adds it as an environment variable during build time.
+//!
+//! This is used so that the python code can detect when the built native module
+//! does not match the source in-tree, helping to detect the case where the
+//! source has been updated but the library hasn't been rebuilt.
+
+use std::path::PathBuf;
+
+use blake2::{Blake2b512, Digest};
+
+fn main() -> Result<(), std::io::Error> {
+    let mut dirs = vec![PathBuf::from("src")];
+
+    let mut paths = Vec::new();
+    while let Some(path) = dirs.pop() {
+        let mut entries = std::fs::read_dir(path)?
+            .map(|res| res.map(|e| e.path()))
+            .collect::<Result<Vec<_>, std::io::Error>>()?;
+
+        entries.sort();
+
+        for entry in entries {
+            if entry.is_dir() {
+                dirs.push(entry)
+            } else {
+                paths.push(entry.to_str().expect("valid rust paths").to_string());
+            }
+        }
+    }
+
+    paths.sort();
+
+    let mut hasher = Blake2b512::new();
+
+    for path in paths {
+        let bytes = std::fs::read(path)?;
+        hasher.update(bytes);
+    }
+
+    let hex_digest = hex::encode(hasher.finalize());
+    println!("cargo:rustc-env=SYNAPSE_RUST_DIGEST={hex_digest}");
+
+    Ok(())
+}
@@ -1,6 +1,12 @@
 use pyo3::prelude::*;

-mod lru_cache;
+/// Returns the hash of all the rust source files at the time it was compiled.
+///
+/// Used by python to detect if the rust library is outdated.
+#[pyfunction]
+fn get_rust_file_digest() -> &'static str {
+    env!("SYNAPSE_RUST_DIGEST")
+}

 /// Formats the sum of two numbers as string.
 #[pyfunction]
@@ -11,9 +17,8 @@ fn sum_as_string(a: usize, b: usize) -> PyResult<String> {

 /// The entry point for defining the Python module.
 #[pymodule]
-fn synapse_rust(py: Python<'_>, m: &PyModule) -> PyResult<()> {
+fn synapse_rust(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
-
-    lru_cache::register_module(py, m)?;
+    m.add_function(wrap_pyfunction!(get_rust_file_digest, m)?)?;
    Ok(())
 }
@@ -1,236 +0,0 @@
-use std::sync::{Arc, Mutex};
-
-use intrusive_collections::{intrusive_adapter, LinkedListAtomicLink};
-use intrusive_collections::{LinkedList, LinkedListLink};
-use lazy_static::lazy_static;
-use log::error;
-use pyo3::prelude::*;
-use pyo3::types::PySet;
-
-/// Called when registering modules with python.
-pub fn register_module(py: Python<'_>, m: &PyModule) -> PyResult<()> {
-    let child_module = PyModule::new(py, "push")?;
-    child_module.add_class::<LruCacheNode>()?;
-    child_module.add_class::<PerCacheLinkedList>()?;
-    child_module.add_function(wrap_pyfunction!(get_global_list, m)?)?;
-
-    m.add_submodule(child_module)?;
-
-    // We need to manually add the module to sys.modules to make `from
-    // synapse.synapse_rust import push` work.
-    py.import("sys")?
-        .getattr("modules")?
-        .set_item("synapse.synapse_rust.lru_cache", child_module)?;
-
-    Ok(())
-}
-
-#[pyclass]
-#[derive(Clone)]
-struct PerCacheLinkedList(Arc<Mutex<LinkedList<LruCacheNodeAdapterPerCache>>>);
-
-#[pymethods]
-impl PerCacheLinkedList {
-    #[new]
-    fn new() -> PerCacheLinkedList {
-        PerCacheLinkedList(Default::default())
-    }
-
-    fn get_back(&self) -> Option<LruCacheNode> {
-        let list = self.0.lock().expect("poisoned");
-        list.back().clone_pointer().map(|n| LruCacheNode(n))
-    }
-}
-
-struct LruCacheNodeInner {
-    per_cache_link: LinkedListAtomicLink,
-    global_list_link: LinkedListAtomicLink,
-    per_cache_list: Arc<Mutex<LinkedList<LruCacheNodeAdapterPerCache>>>,
-    cache: Mutex<Option<PyObject>>,
-    key: PyObject,
-    value: Arc<Mutex<PyObject>>,
-    callbacks: Py<PySet>,
-    memory: usize,
-}
-
-#[pyclass]
-struct LruCacheNode(Arc<LruCacheNodeInner>);
-
-#[pymethods]
-impl LruCacheNode {
-    #[new]
-    fn py_new(
-        cache: PyObject,
-        cache_list: PerCacheLinkedList,
-        key: PyObject,
-        value: PyObject,
-        callbacks: Py<PySet>,
-        memory: usize,
-    ) -> Self {
-        let node = Arc::new(LruCacheNodeInner {
-            per_cache_link: Default::default(),
-            global_list_link: Default::default(),
-            per_cache_list: cache_list.0,
-            cache: Mutex::new(Some(cache)),
-            key,
-            value: Arc::new(Mutex::new(value)),
-            callbacks,
-            memory,
-        });
-
-        GLOBAL_LIST
-            .lock()
-            .expect("posioned")
-            .push_front(node.clone());
-
-        node.per_cache_list
-            .lock()
-            .expect("posioned")
-            .push_front(node.clone());
-
-        LruCacheNode(node)
-    }
-
-    fn add_callbacks(&self, py: Python<'_>, new_callbacks: &PyAny) -> PyResult<()> {
-        if new_callbacks.len()? == 0 {
-            return Ok(());
-        }
-
-        let current_callbacks = self.0.callbacks.as_ref(py);
-
-        for cb in new_callbacks.iter()? {
-            current_callbacks.add(cb?)?;
-        }
-
-        Ok(())
-    }
-
-    fn run_and_clear_callbacks(&self, py: Python<'_>) {
-        let callbacks = self.0.callbacks.as_ref(py);
-
-        if callbacks.is_empty() {
-            return;
-        }
-
-        for callback in callbacks {
-            if let Err(err) = callback.call0() {
-                error!("LruCacheNode callback errored: {err}");
-            }
-        }
-
-        callbacks.clear();
-    }
-
-    fn drop_from_cache(&self) -> PyResult<()> {
-        let cache = self.0.cache.lock().expect("poisoned").take();
-
-        if let Some(cache) = cache {
-            Python::with_gil(|py| cache.call_method1(py, "pop", (&self.0.key, None::<()>)))?;
-        }
-
-        self.drop_from_lists();
-
-        Ok(())
-    }
-
-    fn drop_from_lists(&self) {
-        if self.0.global_list_link.is_linked() {
-            let mut glboal_list = GLOBAL_LIST.lock().expect("poisoned");
-
-            let mut curor_mut = unsafe {
-                // Getting the cursor is unsafe as we need to ensure the list link
-                // belongs to the given list.
-                glboal_list.cursor_mut_from_ptr(Arc::into_raw(self.0.clone()))
-            };
-
-            curor_mut.remove();
-        }
-
-        if self.0.per_cache_link.is_linked() {
-            let mut per_cache_list = self.0.per_cache_list.lock().expect("poisoned");
-
-            let mut curor_mut = unsafe {
-                // Getting the cursor is unsafe as we need to ensure the list link
-                // belongs to the given list.
-                per_cache_list.cursor_mut_from_ptr(Arc::into_raw(self.0.clone()))
-            };
-
-            curor_mut.remove();
-        }
-    }
-
-    fn move_to_front(&self) {
-        if self.0.global_list_link.is_linked() {
-            let mut global_list = GLOBAL_LIST.lock().expect("poisoned");
-
-            let mut curor_mut = unsafe {
-                // Getting the cursor is unsafe as we need to ensure the list link
-                // belongs to the given list.
-                global_list.cursor_mut_from_ptr(Arc::into_raw(self.0.clone()))
-            };
-            curor_mut.remove();
-
-            global_list.push_front(self.0.clone());
-        }
-
-        if self.0.per_cache_link.is_linked() {
-            let mut per_cache_list = self.0.per_cache_list.lock().expect("poisoned");
-
-            let mut curor_mut = unsafe {
-                // Getting the cursor is unsafe as we need to ensure the list link
-                // belongs to the given list.
-                per_cache_list.cursor_mut_from_ptr(Arc::into_raw(self.0.clone()))
-            };
-
-            curor_mut.remove();
-
-            per_cache_list.push_front(self.0.clone());
-        }
-    }
-
-    #[getter]
-    fn key(&self) -> &PyObject {
-        &self.0.key
-    }
-
-    #[getter]
-    fn value(&self) -> PyObject {
-        self.0.value.lock().expect("poisoned").clone()
-    }
-
-    #[setter]
-    fn set_value(&self, value: PyObject) {
-        *self.0.value.lock().expect("poisoned") = value
-    }
-
-    #[getter]
-    fn memory(&self) -> usize {
-        self.0.memory
-    }
-}
-
-#[pyfunction]
-fn get_global_list() -> Vec<LruCacheNode> {
-    let list = GLOBAL_LIST.lock().expect("poisoned");
-
-    let mut vec = Vec::new();
-
-    let mut cursor = list.front();
-
-    while let Some(n) = cursor.clone_pointer() {
-        vec.push(LruCacheNode(n));
-
-        cursor.move_next();
-    }
-
-    vec
-}
-
-intrusive_adapter!(LruCacheNodeAdapterPerCache = Arc<LruCacheNodeInner>: LruCacheNodeInner { per_cache_link: LinkedListLink });
-intrusive_adapter!(LruCacheNodeAdapterGlobal = Arc<LruCacheNodeInner>: LruCacheNodeInner { global_list_link: LinkedListLink });
-
-lazy_static! {
-    static ref GLOBAL_LIST_ADAPTER: LruCacheNodeAdapterGlobal = LruCacheNodeAdapterGlobal::new();
-    static ref GLOBAL_LIST: Arc<Mutex<LinkedList<LruCacheNodeAdapterGlobal>>> =
-        Arc::new(Mutex::new(LinkedList::new(GLOBAL_LIST_ADAPTER.clone())));
-}
@@ -9,8 +9,10 @@
 export PGHOST="localhost"
 POSTGRES_DB_NAME="synapse_full_schema.$$"

-SQLITE_FULL_SCHEMA_OUTPUT_FILE="full.sql.sqlite"
-POSTGRES_FULL_SCHEMA_OUTPUT_FILE="full.sql.postgres"
+SQLITE_SCHEMA_FILE="schema.sql.sqlite"
+SQLITE_ROWS_FILE="rows.sql.sqlite"
+POSTGRES_SCHEMA_FILE="full.sql.postgres"
+POSTGRES_ROWS_FILE="rows.sql.postgres"

 REQUIRED_DEPS=("matrix-synapse" "psycopg2")

@@ -22,7 +24,7 @@ usage() {
  echo "  Username to connect to local postgres instance. The password will be requested"
  echo "  during script execution."
  echo "-c"
-  echo "  CI mode. Enables coverage tracking and prints every command that the script runs."
+  echo "  CI mode. Prints every command that the script runs."
  echo "-o <path>"
  echo "  Directory to output full schema files to."
  echo "-h"
@@ -37,11 +39,6 @@ while getopts "p:co:h" opt; do
    c)
      # Print all commands that are being executed
      set -x
-
-      # Modify required dependencies for coverage
-      REQUIRED_DEPS+=("coverage" "coverage-enable-subprocess")
-
-      COVERAGE=1
      ;;
    o)
      command -v realpath > /dev/null || (echo "The -o flag requires the 'realpath' binary to be installed" && exit 1)
@@ -102,6 +99,7 @@ SQLITE_DB=$TMPDIR/homeserver.db
 POSTGRES_CONFIG=$TMPDIR/postgres.conf

 # Ensure these files are delete on script exit
+# TODO: the trap should also drop the temp postgres DB
 trap 'rm -rf $TMPDIR' EXIT

 cat > "$SQLITE_CONFIG" <<EOF
@@ -147,48 +145,34 @@ python -m synapse.app.homeserver --generate-keys -c "$SQLITE_CONFIG"

 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
 echo "Running db background jobs..."
-synapse/_scripts/update_synapse_database.py --database-config --run-background-updates "$SQLITE_CONFIG"
+synapse/_scripts/update_synapse_database.py --database-config "$SQLITE_CONFIG" --run-background-updates

 # Create the PostgreSQL database.
 echo "Creating postgres database..."
 createdb --lc-collate=C --lc-ctype=C --template=template0 "$POSTGRES_DB_NAME"

-echo "Copying data from SQLite3 to Postgres with synapse_port_db..."
-if [ -z "$COVERAGE" ]; then
-  # No coverage needed
-  synapse/_scripts/synapse_port_db.py --sqlite-database "$SQLITE_DB" --postgres-config "$POSTGRES_CONFIG"
-else
-  # Coverage desired
-  coverage run synapse/_scripts/synapse_port_db.py --sqlite-database "$SQLITE_DB" --postgres-config "$POSTGRES_CONFIG"
-fi
+echo "Running db background jobs..."
+synapse/_scripts/update_synapse_database.py --database-config "$POSTGRES_CONFIG" --run-background-updates
+

 # Delete schema_version, applied_schema_deltas and applied_module_schemas tables
 # Also delete any shadow tables from fts4
-# This needs to be done after synapse_port_db is run
 echo "Dropping unwanted db tables..."
 SQL="
 DROP TABLE schema_version;
 DROP TABLE applied_schema_deltas;
 DROP TABLE applied_module_schemas;
-DROP TABLE event_search_content;
-DROP TABLE event_search_segments;
-DROP TABLE event_search_segdir;
-DROP TABLE event_search_docsize;
-DROP TABLE event_search_stat;
-DROP TABLE user_directory_search_content;
-DROP TABLE user_directory_search_segments;
-DROP TABLE user_directory_search_segdir;
-DROP TABLE user_directory_search_docsize;
-DROP TABLE user_directory_search_stat;
 "
 sqlite3 "$SQLITE_DB" <<< "$SQL"
 psql "$POSTGRES_DB_NAME" -w <<< "$SQL"

-echo "Dumping SQLite3 schema to '$OUTPUT_DIR/$SQLITE_FULL_SCHEMA_OUTPUT_FILE'..."
-sqlite3 "$SQLITE_DB" ".dump" > "$OUTPUT_DIR/$SQLITE_FULL_SCHEMA_OUTPUT_FILE"
+echo "Dumping SQLite3 schema to '$OUTPUT_DIR/$SQLITE_SCHEMA_FILE' and '$OUTPUT_DIR/$SQLITE_ROWS_FILE'..."
+sqlite3 "$SQLITE_DB" ".schema --indent" > "$OUTPUT_DIR/$SQLITE_SCHEMA_FILE"
+sqlite3 "$SQLITE_DB" ".dump --data-only --nosys" > "$OUTPUT_DIR/$SQLITE_ROWS_FILE"

-echo "Dumping Postgres schema to '$OUTPUT_DIR/$POSTGRES_FULL_SCHEMA_OUTPUT_FILE'..."
-pg_dump --format=plain --no-tablespaces --no-acl --no-owner $POSTGRES_DB_NAME | sed -e '/^--/d' -e 's/public\.//g' -e '/^SET /d' -e '/^SELECT /d' > "$OUTPUT_DIR/$POSTGRES_FULL_SCHEMA_OUTPUT_FILE"
+echo "Dumping Postgres schema to '$OUTPUT_DIR/$POSTGRES_SCHEMA_FILE' and '$OUTPUT_DIR/$POSTGRES_ROWS_FILE'..."
+pg_dump --format=plain --schema-only         --no-tablespaces --no-acl --no-owner "$POSTGRES_DB_NAME" | sed -e '/^$/d' -e '/^--/d' -e 's/public\.//g' -e '/^SET /d' -e '/^SELECT /d' > "$OUTPUT_DIR/$POSTGRES_SCHEMA_FILE"
+pg_dump --format=plain --data-only --inserts --no-tablespaces --no-acl --no-owner "$POSTGRES_DB_NAME" | sed -e '/^$/d' -e '/^--/d' -e 's/public\.//g' -e '/^SET /d' -e '/^SELECT /d' > "$OUTPUT_DIR/$POSTGRES_ROWS_FILE"

 echo "Cleaning up temporary Postgres database..."
 dropdb $POSTGRES_DB_NAME
@@ -1 +1,2 @@
 def sum_as_string(a: int, b: int) -> str: ...
+def get_rust_file_digest() -> str: ...
@@ -20,6 +20,8 @@ import json
 import os
 import sys

+from synapse.util.rust import check_rust_lib_up_to_date
+
 # Check that we're not running on an unsupported Python version.
 if sys.version_info < (3, 7):
    print("Synapse requires Python 3.7 or above.")
@@ -78,3 +80,6 @@ if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    from synapse.util.patch_inline_callbacks import do_patch

    do_patch()
+
+
+check_rust_lib_up_to_date()
@@ -67,6 +67,7 @@ from synapse.storage.databases.main.media_repository import (
 )
 from synapse.storage.databases.main.presence import PresenceBackgroundUpdateStore
 from synapse.storage.databases.main.pusher import PusherWorkerStore
+from synapse.storage.databases.main.receipts import ReceiptsBackgroundUpdateStore
 from synapse.storage.databases.main.registration import (
    RegistrationBackgroundUpdateStore,
    find_max_generated_user_id_localpart,
@@ -203,6 +204,7 @@ class Store(
    PushRuleStore,
    PusherWorkerStore,
    PresenceBackgroundUpdateStore,
+    ReceiptsBackgroundUpdateStore,
 ):
    def execute(self, f: Callable[..., R], *args: Any, **kwargs: Any) -> Awaitable[R]:
        return self.db_pool.runInteraction(f.__name__, f, *args, **kwargs)
@@ -32,6 +32,7 @@ from synapse.appservice import ApplicationService
 from synapse.http import get_request_user_agent
 from synapse.http.site import SynapseRequest
 from synapse.logging.opentracing import (
+    SynapseTags,
    active_span,
    force_tracing,
    start_active_span,
@@ -161,6 +162,12 @@ class Auth:
                parent_span.set_tag(
                    "authenticated_entity", requester.authenticated_entity
                )
+                # We tag the Synapse instance name so that it's an easy jumping
+                # off point into the logs. Can also be used to filter for an
+                # instance that is under load.
+                parent_span.set_tag(
+                    SynapseTags.INSTANCE_NAME, self.hs.get_instance_name()
+                )
                parent_span.set_tag("user_id", requester.user.to_string())
                if requester.device_id is not None:
                    parent_span.set_tag("device_id", requester.device_id)
@@ -452,15 +459,6 @@ class Auth:
            )
            raise InvalidClientTokenError("Invalid access token passed.")

-    def get_appservice_by_req(self, request: SynapseRequest) -> ApplicationService:
-        token = self.get_access_token_from_request(request)
-        service = self.store.get_app_service_by_token(token)
-        if not service:
-            logger.warning("Unrecognised appservice access token.")
-            raise InvalidClientTokenError()
-        request.requester = create_requester(service.sender, app_service=service)
-        return service
-
    async def is_server_admin(self, requester: Requester) -> bool:
        """Check if the given user is a local server admin.

@@ -549,8 +549,7 @@ class FederationClientKeysClaimServlet(BaseFederationServerServlet):


 class FederationGetMissingEventsServlet(BaseFederationServerServlet):
-    # TODO(paul): Why does this path alone end with "/?" optional?
-    PATH = "/get_missing_events/(?P<room_id>[^/]*)/?"
+    PATH = "/get_missing_events/(?P<room_id>[^/]*)"

    async def on_POST(
        self,
@@ -45,7 +45,6 @@ from synapse.types import (
    JsonDict,
    StreamKeyType,
    StreamToken,
-    UserID,
    get_domain_from_id,
    get_verify_key_from_cross_signing_key,
 )
@@ -324,8 +323,6 @@ class DeviceHandler(DeviceWorkerHandler):
            self.device_list_updater.incoming_device_list_update,
        )

-        hs.get_distributor().observe("user_left_room", self.user_left_room)
-
        # Whether `_handle_new_device_update_async` is currently processing.
        self._handle_new_device_update_is_processing = False

@@ -569,14 +566,6 @@ class DeviceHandler(DeviceWorkerHandler):
            StreamKeyType.DEVICE_LIST, position, users=[from_user_id]
        )

-    async def user_left_room(self, user: UserID, room_id: str) -> None:
-        user_id = user.to_string()
-        room_ids = await self.store.get_rooms_for_user(user_id)
-        if not room_ids:
-            # We no longer share rooms with this user, so we'll no longer
-            # receive device updates. Mark this in DB.
-            await self.store.mark_remote_user_device_list_as_unsubscribed(user_id)
-
    async def store_dehydrated_device(
        self,
        user_id: str,
@@ -175,6 +175,32 @@ class E2eKeysHandler:
                    user_ids_not_in_cache,
                    remote_results,
                ) = await self.store.get_user_devices_from_cache(query_list)
+
+                # Check that the homeserver still shares a room with all cached users.
+                # Note that this check may be slightly racy when a remote user leaves a
+                # room after we have fetched their cached device list. In the worst case
+                # we will do extra federation queries for devices that we had cached.
+                cached_users = set(remote_results.keys())
+                valid_cached_users = (
+                    await self.store.get_users_server_still_shares_room_with(
+                        remote_results.keys()
+                    )
+                )
+                invalid_cached_users = cached_users - valid_cached_users
+                if invalid_cached_users:
+                    # Fix up results. If we get here, there is either a bug in device
+                    # list tracking, or we hit the race mentioned above.
+                    user_ids_not_in_cache.update(invalid_cached_users)
+                    for invalid_user_id in invalid_cached_users:
+                        remote_results.pop(invalid_user_id)
+                    # This log message may be removed if it turns out it's almost
+                    # entirely triggered by races.
+                    logger.error(
+                        "Devices for %s were cached, but the server no longer shares "
+                        "any rooms with them. The cached device lists are stale.",
+                        invalid_cached_users,
+                    )
+
                for user_id, devices in remote_results.items():
                    user_devices = results.setdefault(user_id, {})
                    for device_id, device in devices.items():
@@ -862,6 +862,9 @@ class FederationEventHandler:
            self._sanity_check_event(event)
        except SynapseError as err:
            logger.warning("Event %s failed sanity check: %s", event_id, err)
+            await self._store.record_event_failed_pull_attempt(
+                event.room_id, event_id, str(err)
+            )
            return

        try:
@@ -897,6 +900,10 @@ class FederationEventHandler:
                    backfilled=backfilled,
                )
        except FederationError as e:
+            await self._store.record_event_failed_pull_attempt(
+                event.room_id, event_id, str(e)
+            )
+
            if e.code == 403:
                logger.warning("Pulled event %s failed history check.", event_id)
            else:
@@ -1057,7 +1064,8 @@ class FederationEventHandler:
        )

        logger.debug(
-            "state_ids returned %i state events, %i auth events",
+            "_get_state_ids_after_missing_prev_event(event_id=%s): state_ids returned %i state events, %i auth events",
+            event_id,
            len(state_event_ids),
            len(auth_event_ids),
        )
@@ -1065,12 +1073,17 @@ class FederationEventHandler:
        # Start by checking events we already have in the DB
        desired_events = set(state_event_ids)
        desired_events.add(event_id)
-        logger.debug("Fetching %i events from cache/store", len(desired_events))
+        logger.debug(
+            "_get_state_ids_after_missing_prev_event(event_id=%s): Fetching %i events from cache/store",
+            event_id,
+            len(desired_events),
+        )
        have_events = await self._store.have_seen_events(room_id, desired_events)

        missing_desired_event_ids = desired_events - have_events
        logger.debug(
-            "We are missing %i events (got %i)",
+            "_get_state_ids_after_missing_prev_event(event_id=%s): We are missing %i events (got %i)",
+            event_id,
            len(missing_desired_event_ids),
            len(have_events),
        )
@@ -1087,7 +1100,11 @@ class FederationEventHandler:
        missing_auth_event_ids.difference_update(
            await self._store.have_seen_events(room_id, missing_auth_event_ids)
        )
-        logger.debug("We are also missing %i auth events", len(missing_auth_event_ids))
+        logger.debug(
+            "_get_state_ids_after_missing_prev_event(event_id=%s): We are also missing %i auth events",
+            event_id,
+            len(missing_auth_event_ids),
+        )

        missing_event_ids = missing_desired_event_ids | missing_auth_event_ids

@@ -1118,10 +1135,17 @@ class FederationEventHandler:
        # TODO: might it be better to have an API which lets us do an aggregate event
        #   request
        if (len(missing_event_ids) * 10) >= len(auth_event_ids) + len(state_event_ids):
-            logger.debug("Requesting complete state from remote")
+            logger.debug(
+                "_get_state_ids_after_missing_prev_event(event_id=%s): Requesting complete state from remote",
+                event_id,
+            )
            await self._get_state_and_persist(destination, room_id, event_id)
        else:
-            logger.debug("Fetching %i events from remote", len(missing_event_ids))
+            logger.debug(
+                "_get_state_ids_after_missing_prev_event(event_id=%s): Fetching %i events from remote",
+                event_id,
+                len(missing_event_ids),
+            )
            await self._get_events_and_persist(
                destination=destination, room_id=room_id, event_ids=missing_event_ids
            )
@@ -1140,8 +1164,9 @@ class FederationEventHandler:
                # This can happen if a remote server claims that the state or
                # auth_events at an event in room A are actually events in room B
                logger.warning(
-                    "Remote server %s claims event %s in room %s is an auth/state "
+                    "_get_state_ids_after_missing_prev_event(event_id=%s):  Remote server %s claims event %s in room %s is an auth/state "
                    "event in room %s",
+                    event_id,
                    destination,
                    state_event_id,
                    metadata.room_id,
@@ -1151,7 +1176,9 @@ class FederationEventHandler:

            if metadata.state_key is None:
                logger.warning(
-                    "Remote server gave us non-state event in state: %s", state_event_id
+                    "_get_state_ids_after_missing_prev_event(event_id=%s): Remote server gave us non-state event in state: %s",
+                    event_id,
+                    state_event_id,
                )
                continue

@@ -1167,6 +1194,14 @@ class FederationEventHandler:
        if not remote_event:
            raise Exception("Unable to get missing prev_event %s" % (event_id,))

+        # Figure out which events we failed to fetch
+        failed_to_fetch = desired_events - event_metadata.keys()
+        # The event_id is part of the `desired_events` but isn't fetched as part
+        # of the `event_metadata` so we remove it here separately if we did find it.
+        have_event_id = await self._store.have_seen_event(room_id, event_id)
+        if have_event_id:
+            failed_to_fetch = failed_to_fetch - {event_id}
+
        # missing state at that event is a warning, not a blocker
        # XXX: this doesn't sound right? it means that we'll end up with incomplete
        #   state.
@@ -1176,7 +1211,7 @@ class FederationEventHandler:
        failed_to_fetch.discard(event_id)
        if failed_to_fetch:
            logger.warning(
-                "Failed to fetch missing state events for %s %s",
+                "_get_state_ids_after_missing_prev_event(event_id=%s): Failed to fetch missing state events %s",
                event_id,
                failed_to_fetch,
            )
@@ -752,20 +752,12 @@ class EventCreationHandler:
        if builder.type == EventTypes.Member:
            membership = builder.content.get("membership", None)
            if membership == Membership.JOIN:
-                return await self._is_server_notices_room(builder.room_id)
+                return await self.store.is_server_notice_room(builder.room_id)
            elif membership == Membership.LEAVE:
                # the user is always allowed to leave (but not kick people)
                return builder.state_key == requester.user.to_string()
        return False

-    async def _is_server_notices_room(self, room_id: str) -> bool:
-        if self.config.servernotices.server_notices_mxid is None:
-            return False
-        is_server_notices_room = await self.store.check_local_user_in_room(
-            user_id=self.config.servernotices.server_notices_mxid, room_id=room_id
-        )
-        return is_server_notices_room
-
    async def assert_accepted_privacy_policy(self, requester: Requester) -> None:
        """Check if a user has accepted the privacy policy

@@ -837,7 +837,7 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                old_membership == Membership.INVITE
                and effective_membership_state == Membership.LEAVE
            ):
-                is_blocked = await self._is_server_notice_room(room_id)
+                is_blocked = await self.store.is_server_notice_room(room_id)
                if is_blocked:
                    raise SynapseError(
                        HTTPStatus.FORBIDDEN,
@@ -1617,14 +1617,6 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):

        return False

-    async def _is_server_notice_room(self, room_id: str) -> bool:
-        if self._server_notices_mxid is None:
-            return False
-        is_server_notices_room = await self.store.check_local_user_in_room(
-            user_id=self._server_notices_mxid, room_id=room_id
-        )
-        return is_server_notices_room
-

 class RoomMemberMasterHandler(RoomMemberHandler):
    def __init__(self, hs: "HomeServer"):
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Eric Eastwood	dab846568d	Merge branch 'develop' into madlittlemods/event_id_always_failed_to_fetch Conflicts: synapse/handlers/federation_event.py	2022-09-14 18:14:17 -05:00
Eric Eastwood	957e3d74fc	Keep track when we try and fail to process a pulled event (#13589 ) We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)	2022-09-14 13:57:50 -05:00
Patrick Cloke	666ae87729	Update event push action and receipt tables to support threads. (#13753 ) Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` tables. This will notifications to be segmented by the thread in a future pull request. The `thread_id` column stores the root event ID or the special value `"main"`. The `thread_id` column for `event_push_actions` and `event_push_summary` is backfilled with `"main"` for all existing rows. New entries into `event_push_actions` and `event_push_actions_staging` will get the proper thread ID. `receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar, except `NULL` is a special value meaning the receipt is "unthreaded". See MSC3771 and MSC3773 for where this data will be useful.	2022-09-14 17:11:16 +00:00
Patrick Cloke	f2d12ccabe	Use partial indices on SQLIte. (#13802 ) Partial indices have been supported since SQLite 3.8, but Synapse now requires >= 3.27, so we can enable support for them. This requires rebuilding previous indices which were partial on PostgreSQL, but not on SQLite.	2022-09-14 12:01:42 -04:00
reivilibre	6302753012	Deduplicate `is_server_notices_room`. (#13780 )	2022-09-14 15:53:18 +00:00
reivilibre	cf65433de2	Fix a memory leak when running the unit tests. (#13798 )	2022-09-14 15:29:05 +00:00
Quentin Gliech	eaed4e6113	Remove unused method in `synapse.api.auth.Auth`. (#13795 ) Clean-up from `b19060a29b` (#13094) and `73af10f419` (#13093) which removed all callers.	2022-09-14 10:33:54 -04:00
David Robertson	51a77e990b	Remove incorrect migration file from `state` logical DB (#13788 ) * Remove incorrect migration file from `state` logical DB The table `ex_outlier_stream` is part of the `main` logical DB; it should not have been created in the `state` logical DB. We remove this migration now as a tidy-up. Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new migration, because some (most) instances of Synapse host both of these logical DBs on the same DB cluster. * Changelog	2022-09-14 14:16:12 +01:00
Sean Quah	c73774467e	Fix bug in device list caching when remote users leave rooms (#13749 ) When a remote user leaves the last room shared with the homeserver, we have to mark their device list as unsubscribed, otherwise we would hold on to a stale device list in our cache. Crucially, the device list would remain cached even after the remote user rejoined the room, which could lead to E2EE failures until the next change to the remote user's device list. Fixes #13651. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-09-14 10:42:57 +01:00
reivilibre	21687ec189	Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests. (#13789 ) * Don't accept a trailing slash on the end of /get_missing_events * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>	2022-09-14 09:28:12 +01:00
Mathieu Velten	12dacecabd	Make sequence `cache_invalidation_stream_seq` begin at `2` (#13766 ) Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>	2022-09-13 16:14:28 +02:00
Erik Johnston	9772e362aa	Merge branch 'master' into develop	2022-09-13 12:11:53 +01:00
David Robertson	b60d47ab2c	Updates to the schema dump script (#13770 )	2022-09-13 10:53:11 +01:00
David Robertson	540afb0bfc	Simplify CI tests DAG (#13784 ) * Simplify CI tests DAG * Changelog	2022-09-13 10:17:23 +01:00
Richard van der Hoff	41df25bbbd	installation.md: require libpq on M1 macs (#13480 )	2022-09-13 09:01:21 +00:00
Erik Johnston	80bb098d87	Fixup changelog	2022-09-13 09:55:10 +01:00
Erik Johnston	4b678b20a2	1.67.0	2022-09-13 09:20:28 +01:00
Nick Mills-Barrett	cdbb641232	Add receipts event stream ordering (#13703 )	2022-09-13 08:16:37 +01:00
Mathieu Velten	fa2f3d8d0c	Fix GHA skippable syntax (#13778 ) Signed-off-by: Mathieu Velten <mathieuv@matrix.org>	2022-09-12 17:31:23 +00:00
Brendan Abolivier	7571337445	Fix typo in ratelimiting documentation (#13727 )	2022-09-12 14:11:18 +01:00
Erik Johnston	dd7484b562	Fix CI on non-PR builds (#13769 ) Mark cargo-test as skippable since it only runs on Rust code change.	2022-09-12 13:26:33 +01:00
Nick Mills-Barrett	da41a7cd61	Remove check current state membership up to date (#13745 ) * Remove checks for membership column in current_state_events * Add schema script to force through the `current_state_events_membership` background job Contributed by Nick @ Beeper (@fizzadar).	2022-09-12 12:58:33 +01:00
Erik Johnston	ebfeac7c5d	Check if Rust lib needs rebuilding. (#13759 ) This protects against the common mistake of failing to remember to rebuild Rust code after making changes.	2022-09-12 10:03:42 +00:00
Nick Mills-Barrett	4c4889cac0	Concurrently collect room unread counts for push badges (#13765 ) Most of the time this function is heavily cached, but when that isn't the case fetching the counts room by room slows down push delivery on users with many (thousands) of rooms. Signed off by Nick @ Beeper.	2022-09-09 19:00:21 +01:00
Eric Eastwood	a911ffb42c	Tag trace with instance name (#13761 ) We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load. As suggested by @clokep and @reivilibre in, - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578	2022-09-09 11:31:37 -05:00
Eric Eastwood	f694bb71b7	Strip number suffix from instance name to consolidate services that traces are spread over (#13729 ) The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985 Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through. With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇 Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.	2022-09-09 11:30:06 -05:00
David Robertson	9d11842562	Also cite upgrade notes	2022-09-08 12:04:29 +01:00
David Robertson	a7c71686ca	Add minimum version bump for sqlite to the release notes (#13742 ) * Notify that SQLite min version will be bumped * Mention in upgrade notes Co-authored-by: reivilibre <oliverw@matrix.org>	2022-09-08 12:00:03 +01:00
Erik Johnston	ec2fe7bb53	Fixup changelog	2022-09-06 13:04:22 +01:00
Erik Johnston	571f565c1f	Update changelog	2022-09-06 09:25:32 +01:00
Erik Johnston	cdf7fb737b	1.67.0rc1	2022-09-06 09:01:26 +01:00
Eric Eastwood	b393f71374	Add changelog	2022-07-11 17:04:20 -05:00
Eric Eastwood	3d16e72918	Different comments	2022-07-11 15:38:43 -05:00
Eric Eastwood	cc32634cd0	Fix event_id always being marked as failed to fetch when fetching state for it Discovered while working on https://github.com/matrix-org/synapse/pull/13205	2022-07-11 15:27:49 -05:00
				`@@ -1 +0,0 @@`
				Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
				`@@ -0,0 +1 @@`
				Prevent `Failed to fetch missing state events...` warning being logged every time we tried to fetch `state_ids` for a missing `prev_event`.
				`@@ -0,0 +1 @@`
				Note that `libpq` is required on ARM-based Macs.
				`@@ -1 +0,0 @@`
				`Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process.`
				`@@ -1 +0,0 @@`
				Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database.
				`@@ -1 +0,0 @@`
				`Add experimental configuration option to allow disabling legacy Prometheus metric names.`
				`@@ -1 +0,0 @@`
				Fix bug that user cannot `/forget` rooms after the last member has left the room.
				`@@ -1 +0,0 @@`
				Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar).
				`@@ -1 +0,0 @@`
				`Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar).`