WIP: Type TreeCache

.ci/scripts/test_old_deps.sh

@@ -69,7 +69,7 @@ with open('pyproject.toml', 'w') as f:
 "
 python3 -c "$REMOVE_DEV_DEPENDENCIES"
 
-pipx install poetry==1.1.14
+pipx install poetry==1.1.12
 ~/.local/bin/poetry lock
 
 echo "::group::Patched pyproject.toml"

.git-blame-ignore-revs

@@ -1,16 +1,3 @@
-# Commits in this file will be removed from GitHub blame results.
-#
-# To use this file locally, use:
-#   git blame --ignore-revs-file="path/to/.git-blame-ignore-revs" <files>
-#
-# or configure the `blame.ignoreRevsFile` option in your git config.
-#
-# If ignoring a pull request that was not squash merged, only the merge
-# commit needs to be put here. Child commits will be resolved from it.
-
-# Run black (#3679).
-8b3d9b6b199abb87246f982d5db356f1966db925
-
 # Black reformatting (#5482).
 32e7c9e7f20b57dd081023ac42d6931a8da9b3a3
 

.github/workflows/twisted_trunk.yml

@@ -127,12 +127,12 @@ jobs:
         run: |
           set -x
           DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx
-          pipx install poetry==1.1.14
+          pipx install poetry==1.1.12
 
           poetry remove -n twisted
           poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk
           poetry lock --no-update
-          # NOT IN 1.1.14 poetry lock --check
+          # NOT IN 1.1.12 poetry lock --check
         working-directory: synapse
 
       - run: |

CHANGES.md

@@ -3,25 +3,6 @@ Synapse vNext
 
 As of this release, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. For more information, see the [upgrade notes](https://matrix-org.github.io/synapse/v1.64/upgrade.html#upgrading-to-v1640).
 
-
-Synapse 1.63.1 (2022-07-20)
-===========================
-
-Bugfixes
---------
-
-- Fix a bug introduced in Synapse 1.63.0 where push actions were incorrectly calculated for appservice users. This caused performance issues on servers with large numbers of appservices. ([\#13332](https://github.com/matrix-org/synapse/issues/13332))
-
-
-Synapse 1.63.0 (2022-07-19)
-===========================
-
-Improved Documentation
-----------------------
-
-- Clarify that homeserver server names are included in the reported data when the `report_stats` config option is enabled. ([\#13321](https://github.com/matrix-org/synapse/issues/13321))
-
-
 Synapse 1.63.0rc1 (2022-07-12)
 ==============================
 
@@ -30,7 +11,7 @@ Features
 
 - Add a rate limit for local users sending invites. ([\#13125](https://github.com/matrix-org/synapse/issues/13125))
 - Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of `/publicRooms` by room type. ([\#13031](https://github.com/matrix-org/synapse/issues/13031))
-- Improve validation logic in the account data REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
+- Improve validation logic in Synapse's REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
 
 
 Bugfixes
@@ -58,7 +39,7 @@ Improved Documentation
 - Add an explanation of the `--report-stats` argument to the docs. ([\#13029](https://github.com/matrix-org/synapse/issues/13029))
 - Add a helpful example bash script to the contrib directory for creating multiple worker configuration files of the same type. Contributed by @villepeh. ([\#13032](https://github.com/matrix-org/synapse/issues/13032))
 - Add missing links to config options. ([\#13166](https://github.com/matrix-org/synapse/issues/13166))
-- Add documentation for homeserver usage statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
+- Add documentation for anonymised homeserver statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
 - Add documentation for the existing `databases` option in the homeserver configuration manual. ([\#13212](https://github.com/matrix-org/synapse/issues/13212))
 - Clean up references to sample configuration and redirect users to the configuration manual instead. ([\#13077](https://github.com/matrix-org/synapse/issues/13077), [\#13139](https://github.com/matrix-org/synapse/issues/13139))
 - Document how the Synapse team does reviews. ([\#13132](https://github.com/matrix-org/synapse/issues/13132))
@@ -143,7 +124,7 @@ Bugfixes
 - Update [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786) implementation to check `state_key`. ([\#12939](https://github.com/matrix-org/synapse/issues/12939))
 - Fix a bug introduced in Synapse 1.58 where Synapse would not report full version information when installed from a git checkout. This is a best-effort affair and not guaranteed to be stable. ([\#12973](https://github.com/matrix-org/synapse/issues/12973))
 - Fix a bug introduced in Synapse 1.60 where Synapse would fail to start if the `sqlite3` module was not available. ([\#12979](https://github.com/matrix-org/synapse/issues/12979))
-- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
+- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced 
   in Synapse v1.41.0. ([\#12991](https://github.com/matrix-org/synapse/issues/12991))
 - Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. ([\#13018](https://github.com/matrix-org/synapse/issues/13018))
 - Fix a bug introduced in Synapse 1.58 where profile requests for a malformed user ID would ccause an internal error. Synapse now returns 400 Bad Request in this situation. ([\#13041](https://github.com/matrix-org/synapse/issues/13041))

changelog.d/12942.misc

@@ -1 +0,0 @@
-Use lower isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper.

changelog.d/13038.feature

@@ -1 +0,0 @@
-Provide more info why we don't have any thumbnails to serve.

changelog.d/13172.misc

@@ -1 +0,0 @@
-Always use a version of canonicaljson that supports the C implementation of frozendict.

changelog.d/13205.feature

@@ -1 +0,0 @@
-Allow pagination from remote event after discovering it from MSC3030 `/timestamp_to_event`.

changelog.d/13258.misc

@@ -1 +0,0 @@
-Fix spurious warning when fetching state after a missing prev event.

changelog.d/13271.doc

@@ -1 +0,0 @@
-Add another `contrib` script to help set up worker processes. Contributed by @villepeh.

changelog.d/13276.feature

@@ -1 +0,0 @@
-Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttle additional joins if that rate grows too large.

changelog.d/13281.misc

@@ -1 +0,0 @@
-Don't pull out the full state when creating an event.

changelog.d/13285.misc

@@ -1 +0,0 @@
-Upgrade from Poetry 1.1.14 to 1.1.12, to fix bugs when locking packages.

changelog.d/13296.bugfix

@@ -1 +0,0 @@
-Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced.

changelog.d/13297.misc

@@ -1 +0,0 @@
-Use `HTTPStatus` constants in place of literals in tests.

changelog.d/13299.misc

@@ -1 +0,0 @@
-Improve performance of query  `_get_subset_users_in_room_with_profiles`.

changelog.d/13300.misc

@@ -1 +0,0 @@
-Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`.

changelog.d/13303.misc

@@ -1 +0,0 @@
-Remove unnecessary `json.dumps` from tests.

changelog.d/13307.misc

@@ -1 +0,0 @@
-Don't pull out the full state when creating an event.

changelog.d/13308.misc

@@ -1 +0,0 @@
-Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).

changelog.d/13310.misc

@@ -1 +0,0 @@
-Reduce memory usage of sending dummy events.

changelog.d/13311.misc

@@ -1 +0,0 @@
-Prevent formatting changes of [#3679](https://github.com/matrix-org/synapse/pull/3679) from appearing in `git blame`.

changelog.d/13313.misc

@@ -1 +0,0 @@
-Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries.

changelog.d/13314.doc

@@ -1 +0,0 @@
-Add notes when config options where changed. Contributed by @behrmann.

changelog.d/13318.misc

@@ -1 +0,0 @@
-Validate federation destinations and log an error if a destination is invalid.

changelog.d/13320.misc

@@ -1 +0,0 @@
-Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation.

changelog.d/13323.misc

@@ -1 +0,0 @@
-Reduce memory usage of state caches.

changelog.d/13324.misc

@@ -1 +0,0 @@
-Reduce the amount of state we store in the `state_cache`.

changelog.d/13326.removal

@@ -1 +0,0 @@
-Stop builindg `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life.

changelog.d/13328.misc

@@ -1 +0,0 @@
-Add missing type hints to open tracing module.

changelog.d/13329.misc

@@ -1 +0,0 @@
-Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@fizzadar).

changelog.d/13333.doc

@@ -1 +0,0 @@
-Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63.

changelog.d/13338.doc

@@ -1 +0,0 @@
-Mention that BuildKit is needed when building Docker images for tests.

changelog.d/13342.misc

@@ -1 +0,0 @@
-When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics.

changelog.d/13345.misc

@@ -1 +0,0 @@
-Add missing type hints to open tracing module.

changelog.d/13349.misc

@@ -1 +0,0 @@
-Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@fizzadar).

changelog.d/13352.bugfix

@@ -1 +0,0 @@
-Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`.

changelog.d/13354.misc

@@ -1 +0,0 @@
-Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known.

changelog.d/13362.misc

@@ -1 +0,0 @@
-Add missing type hints to open tracing module.

changelog.d/13395.misc

@@ -1 +0,0 @@
-WIP: stateres debug script.

contrib/workers-bash-scripts/create-multiple-stream-writers.md

@@ -1,145 +0,0 @@
-# Creating multiple stream writers with a bash script
-
-This script creates multiple [stream writer](https://github.com/matrix-org/synapse/blob/develop/docs/workers.md#stream-writers) workers.
-
-Stream writers require both replication and HTTP listeners.
-
-It also prints out the example lines for Synapse main configuration file.
-
-Remember to route necessary endpoints directly to a worker associated with it.
-
-If you run the script as-is, it will create workers with the replication listener starting from port 8034 and another, regular http listener starting from 8044. If you don't need all of the stream writers listed in the script, just remove them from the ```STREAM_WRITERS``` array.
-
-```sh
-#!/bin/bash
-
-# Start with these replication and http ports.
-# The script loop starts with the exact port and then increments it by one.
-REP_START_PORT=8034
-HTTP_START_PORT=8044
-
-# Stream writer workers to generate. Feel free to add or remove them as you wish.
-# Event persister ("events") isn't included here as it does not require its
-# own HTTP listener.
-
-STREAM_WRITERS+=( "presence" "typing" "receipts" "to_device" "account_data" )
-
-NUM_WRITERS=$(expr ${#STREAM_WRITERS[@]})
-
-i=0
-
-while [ $i -lt "$NUM_WRITERS" ]
-do
-cat << EOF > ${STREAM_WRITERS[$i]}_stream_writer.yaml
-worker_app: synapse.app.generic_worker
-worker_name: ${STREAM_WRITERS[$i]}_stream_writer
-
-# The replication listener on the main synapse process.
-worker_replication_host: 127.0.0.1
-worker_replication_http_port: 9093
-
-worker_listeners:
-  - type: http
-    port: $(expr $REP_START_PORT + $i)
-    resources:
-      - names: [replication]
-
-  - type: http
-    port: $(expr $HTTP_START_PORT + $i)
-    resources:
-      - names: [client]
-
-worker_log_config: /etc/matrix-synapse/stream-writer-log.yaml
-EOF
-HOMESERVER_YAML_INSTANCE_MAP+=$"  ${STREAM_WRITERS[$i]}_stream_writer:
-    host: 127.0.0.1
-    port: $(expr $REP_START_PORT + $i)
-"
-
-HOMESERVER_YAML_STREAM_WRITERS+=$"  ${STREAM_WRITERS[$i]}: ${STREAM_WRITERS[$i]}_stream_writer
-"
-
-((i++))
-done
-
-cat << EXAMPLECONFIG
-# Add these lines to your homeserver.yaml.
-# Don't forget to configure your reverse proxy and
-# necessary endpoints to their respective worker.
-
-# See https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
-# for more information.
-
-# Remember: Under NO circumstances should the replication
-# listener be exposed to the public internet;
-# it has no authentication and is unencrypted.
-
-instance_map:
-$HOMESERVER_YAML_INSTANCE_MAP
-stream_writers:
-$HOMESERVER_YAML_STREAM_WRITERS
-EXAMPLECONFIG
-```
-
-Copy the code above save it to a file ```create_stream_writers.sh``` (for example).
-
-Make the script executable by running ```chmod +x create_stream_writers.sh```.
-
-## Run the script to create workers and print out a sample configuration
-
-Simply run the script to create YAML files in the current folder and print out the required configuration for ```homeserver.yaml```.
-
-```console
-$ ./create_stream_writers.sh
-
-# Add these lines to your homeserver.yaml.
-# Don't forget to configure your reverse proxy and
-# necessary endpoints to their respective worker.
-
-# See https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
-# for more information
-
-# Remember: Under NO circumstances should the replication
-# listener be exposed to the public internet;
-# it has no authentication and is unencrypted.
-
-instance_map:
-  presence_stream_writer:
-    host: 127.0.0.1
-    port: 8034
-  typing_stream_writer:
-    host: 127.0.0.1
-    port: 8035
-  receipts_stream_writer:
-    host: 127.0.0.1
-    port: 8036
-  to_device_stream_writer:
-    host: 127.0.0.1
-    port: 8037
-  account_data_stream_writer:
-    host: 127.0.0.1
-    port: 8038
-
-stream_writers:
-  presence: presence_stream_writer
-  typing: typing_stream_writer
-  receipts: receipts_stream_writer
-  to_device: to_device_stream_writer
-  account_data: account_data_stream_writer
-```
-
-Simply copy-and-paste the output to an appropriate place in your Synapse main configuration file.
-
-## Write directly to Synapse configuration file
-
-You could also write the output directly to homeserver main configuration file. **This, however, is not recommended** as even a small typo (such as replacing >> with >) can erase the entire ```homeserver.yaml```. 
-
-If you do this, back up your original configuration file first:
-
-```console
-# Back up homeserver.yaml first
-cp /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/homeserver.yaml.bak 
-
-# Create workers and write output to your homeserver.yaml
-./create_stream_writers.sh >> /etc/matrix-synapse/homeserver.yaml 
-```

contrib/workers-bash-scripts/create-multiple-workers.md

@@ -1,4 +1,4 @@
-# Creating multiple generic workers with a bash script
+# Creating multiple workers with a bash script
 
 Setting up multiple worker configuration files manually can be time-consuming.
 You can alternatively create multiple worker configuration files with a simple `bash` script. For example:

debian/changelog

@@ -1,17 +1,3 @@
-matrix-synapse-py3 (1.63.1) stable; urgency=medium
-
-  * New Synapse release 1.63.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 20 Jul 2022 13:36:52 +0100
-
-matrix-synapse-py3 (1.63.0) stable; urgency=medium
-
-  * Clarify that homeserver server names are included in the data reported
-    by opt-in server stats reporting (`report_stats` homeserver config option).
-  * New Synapse release 1.63.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 19 Jul 2022 14:42:24 +0200
-
 matrix-synapse-py3 (1.63.0~rc1) stable; urgency=medium
 
   * New Synapse release 1.63.0rc1.

debian/matrix-synapse-py3.postinst

@@ -31,7 +31,7 @@ EOF
 # This file is autogenerated, and will be recreated on upgrade if it is deleted.
 # Any changes you make will be preserved.
 
-# Whether to report homeserver usage statistics.
+# Whether to report anonymized homeserver usage statistics.
 report_stats: false
 EOF
     fi

debian/po/templates.pot

@@ -37,7 +37,7 @@ msgstr ""
 #. Type: boolean
 #. Description
 #: ../templates:2001
-msgid "Report homeserver usage statistics?"
+msgid "Report anonymous statistics?"
 msgstr ""
 
 #. Type: boolean
@@ -45,11 +45,11 @@ msgstr ""
 #: ../templates:2001
 msgid ""
 "Developers of Matrix and Synapse really appreciate helping the project out "
-"by reporting homeserver usage statistics from this homeserver. Your "
-"homeserver's server name, along with very basic aggregate data (e.g. "
-"number of users) will be reported. But it helps track the growth of the "
-"Matrix community, and helps in making Matrix a success, as well as to "
-"convince other networks that they should peer with Matrix."
+"by reporting anonymized usage statistics from this homeserver. Only very "
+"basic aggregate data (e.g. number of users) will be reported, but it helps "
+"track the growth of the Matrix community, and helps in making Matrix a "
+"success, as well as to convince other networks that they should peer with "
+"Matrix."
 msgstr ""
 
 #. Type: boolean

debian/templates

@@ -10,13 +10,12 @@ _Description: Name of the server:
 Template: matrix-synapse/report-stats
 Type: boolean
 Default: false
-_Description: Report homeserver usage statistics?
+_Description: Report anonymous statistics?
  Developers of Matrix and Synapse really appreciate helping the
- project out by reporting homeserver usage statistics from this
- homeserver. Your homeserver's server name, along with very basic
- aggregate data (e.g. number of users) will be reported. But it
- helps track the growth of the Matrix community, and helps in
- making Matrix a success, as well as to convince other networks
- that they should peer with Matrix.
+ project out by reporting anonymized usage statistics from this
+ homeserver. Only very basic aggregate data (e.g. number of users)
+ will be reported, but it helps track the growth of the Matrix
+ community, and helps in making Matrix a success, as well as to
+ convince other networks that they should peer with Matrix.
  .
  Thank you.

docker/Dockerfile

@@ -45,7 +45,7 @@ RUN \
 
 # We install poetry in its own build stage to avoid its dependencies conflicting with
 # synapse's dependencies.
-# We use a specific commit from poetry's master branch instead of our usual 1.1.14,
+# We use a specific commit from poetry's master branch instead of our usual 1.1.12,
 # to incorporate fixes to some bugs in `poetry export`. This commit corresponds to
 #    https://github.com/python-poetry/poetry/pull/5156 and
 #    https://github.com/python-poetry/poetry/issues/5141 ;

docker/Dockerfile-workers

@@ -1,4 +1,3 @@
-# syntax=docker/dockerfile:1
 # Inherit from the official Synapse docker image
 ARG SYNAPSE_VERSION=latest
 FROM matrixdotorg/synapse:$SYNAPSE_VERSION

docker/README-testing.md

@@ -22,10 +22,6 @@ Consult the [contributing guide][guideComplementSh] for instructions on how to u
 Under some circumstances, you may wish to build the images manually.
 The instructions below will lead you to doing that.
 
-Note that these images can only be built using [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/),
-therefore BuildKit needs to be enabled when calling `docker build`. This can be done by
-setting `DOCKER_BUILDKIT=1` in your environment.
-
 Start by building the base Synapse docker image. If you wish to run tests with the latest
 release of Synapse, instead of your current checkout, you can skip this step. From the
 root of the repository:

docker/complement/Dockerfile

@@ -1,4 +1,3 @@
-# syntax=docker/dockerfile:1
 # This dockerfile builds on top of 'docker/Dockerfile-workers' in matrix-org/synapse
 # by including a built-in postgres instance, as well as setting up the homeserver so
 # that it is ready for testing via Complement.

docker/complement/conf/workers-shared-extra.yaml.j2

@@ -67,10 +67,6 @@ rc_joins:
     per_second: 9999
     burst_count: 9999
 
-rc_joins_per_room:
-    per_second: 9999
-    burst_count: 9999
-
 rc_3pid_validation:
   per_second: 1000
   burst_count: 1000

docs/SUMMARY.md

@@ -68,7 +68,7 @@
       - [Federation](usage/administration/admin_api/federation.md)
     - [Manhole](manhole.md)
     - [Monitoring](metrics-howto.md)
-      - [Reporting Homeserver Usage Statistics](usage/administration/monitoring/reporting_homeserver_usage_statistics.md)
+      - [Reporting Anonymised Statistics](usage/administration/monitoring/reporting_anonymised_statistics.md)
     - [Understanding Synapse Through Grafana Graphs](usage/administration/understanding_synapse_through_grafana_graphs.md)
     - [Useful SQL for Admins](usage/administration/useful_sql_for_admins.md)
     - [Database Maintenance Tools](usage/administration/database_maintenance_tools.md)

docs/development/dependencies.md

@@ -237,28 +237,3 @@ poetry run pip install build && poetry run python -m build
 because [`build`](https://github.com/pypa/build) is a standardish tool which
 doesn't require poetry. (It's what we use in CI too). However, you could try
 `poetry build` too.
-
-
-# Troubleshooting
-
-## Check the version of poetry with `poetry --version`.
-
-At the time of writing, the 1.2 series is beta only. We have seen some examples
-where the lockfiles generated by 1.2 prereleasese aren't interpreted correctly
-by poetry 1.1.x. For now, use poetry 1.1.14, which includes a critical
-[change](https://github.com/python-poetry/poetry/pull/5973) needed to remain
-[compatible with PyPI](https://github.com/pypi/warehouse/pull/11775).
-
-It can also be useful to check the version of `poetry-core` in use. If you've
-installed `poetry` with `pipx`, try `pipx runpip poetry list | grep poetry-core`.
-
-## Clear caches: `poetry cache clear --all pypi`.
-
-Poetry caches a bunch of information about packages that isn't readily available
-from PyPI. (This is what makes poetry seem slow when doing the first
-`poetry install`.) Try `poetry cache list` and `poetry cache clear --all
-<name of cache>` to see if that fixes things.
-
-## Try `--verbose` or `--dry-run` arguments.
-
-Sometimes useful to see what poetry's internal logic is.

docs/upgrade.md

@@ -104,25 +104,6 @@ minimum, a `notif_from` setting.)
 Specifying an `email` setting under `account_threepid_delegates` will now cause
 an error at startup.
 
-## Changes to the event replication streams
-
-Synapse now includes a flag indicating if an event is an outlier when
-replicating it to other workers. This is a forwards- and backwards-incompatible
-change: v1.63 and workers cannot process events replicated by v1.64 workers, and
-vice versa.
-
-Once all workers are upgraded to v1.64 (or downgraded to v1.63), event
-replication will resume as normal.
-
-## frozendict release
-
-[frozendict 2.3.3](https://github.com/Marco-Sulla/python-frozendict/releases/tag/v2.3.3)
-has recently been released, which fixes a memory leak that occurs during `/sync`
-requests. We advise server administrators who installed Synapse via pip to upgrade
-frozendict with `pip install --upgrade frozendict`. The Docker image
-`matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` already
-include the updated library.
-
 # Upgrading to v1.62.0
 
 ## New signatures for spam checker callbacks

docs/usage/administration/monitoring/reporting_anonymised_statistics.md

@@ -1,11 +1,11 @@
-# Reporting Homeserver Usage Statistics
+# Reporting Anonymised Statistics
 
 When generating your Synapse configuration file, you are asked whether you
-would like to report usage statistics to Matrix.org. These statistics
+would like to report anonymised statistics to Matrix.org. These statistics
 provide the foundation a glimpse into the number of Synapse homeservers
 participating in the network, as well as statistics such as the number of
 rooms being created and messages being sent. This feature is sometimes
-affectionately called "phone home" stats. Reporting
+affectionately called "phone-home" stats. Reporting
 [is optional](../../configuration/config_documentation.md#report_stats)
 and the reporting endpoint
 [can be configured](../../configuration/config_documentation.md#report_stats_endpoint),
@@ -21,9 +21,9 @@ The following statistics are sent to the configured reporting endpoint:
 
 | Statistic Name             | Type   | Description                                                                                                                                                                                                                                                                                     |
 |----------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `homeserver`               | string | The homeserver's server name.                                                                                                                                                                                                                                                                   |
 | `memory_rss`               | int    | The memory usage of the process (in kilobytes on Unix-based systems, bytes on MacOS).                                                                                                                                                                                                           |
 | `cpu_average`              | int    | CPU time in % of a single core (not % of all cores).                                                                                                                                                                                                                                            |              
+| `homeserver`               | string | The homeserver's server name.                                                                                                                                                                                                                                                                   |
 | `server_context`           | string | An arbitrary string used to group statistics from a set of homeservers.                                                                                                                                                                                                                         |
 | `timestamp`                | int    | The current time, represented as the number of seconds since the epoch.                                                                                                                                                                                                                         |                 
 | `uptime_seconds`           | int    | The number of seconds since the homeserver was last started.                                                                                                                                                                                                                                    |

docs/usage/configuration/config_documentation.md

@@ -239,8 +239,6 @@ If this option is provided, it parses the given yaml to json and
 serves it on `/.well-known/matrix/client` endpoint
 alongside the standard properties.
 
-*Added in Synapse 1.62.0.*
-
 Example configuration:
 ```yaml
 extra_well_known_client_content : 
@@ -1157,9 +1155,6 @@ Caching can be configured through the following sub-options:
   with intermittent connections, at the cost of higher memory usage.
   A value of zero means that sync responses are not cached.
   Defaults to 2m.
-
-  *Changed in Synapse 1.62.0*: The default was changed from 0 to 2m.
-
 * `cache_autotuning` and its sub-options `max_cache_memory_usage`, `target_cache_memory_usage`, and
    `min_cache_ttl` work in conjunction with each other to maintain a balance between cache memory 
    usage and cache entry availability. You must be using [jemalloc](https://github.com/matrix-org/synapse#help-synapse-is-slow-and-eats-all-my-ramcpu) 
@@ -1476,25 +1471,6 @@ rc_joins:
     per_second: 0.03
     burst_count: 12
 ```
----
-### `rc_joins_per_room`
-
-This option allows admins to ratelimit joins to a room based on the number of recent
-joins (local or remote) to that room. It is intended to mitigate mass-join spam
-waves which target multiple homeservers.
-
-By default, one join is permitted to a room every second, with an accumulating
-buffer of up to ten instantaneous joins.
-
-Example configuration (default values):
-```yaml
-rc_joins_per_room:
-  per_second: 1
-  burst_count: 10
-```
-
-_Added in Synapse 1.64.0._
-
 ---
 ### `rc_3pid_validation`
 
@@ -1528,10 +1504,6 @@ The `rc_invites.per_user` limit applies to the *receiver* of the invite, rather
 sender, meaning that a `rc_invite.per_user.burst_count` of 5 mandates that a single user
 cannot *receive* more than a burst of 5 invites at a time.
 
-In contrast, the `rc_invites.per_issuer` limit applies to the *issuer* of the invite, meaning that a `rc_invite.per_issuer.burst_count` of 5 mandates that single user cannot *send* more than a burst of 5 invites at a time.
-
-_Changed in version 1.63:_ added the `per_issuer` limit.
-
 Example configuration:
 ```yaml
 rc_invites:
@@ -1541,11 +1513,7 @@ rc_invites:
   per_user:
     per_second: 0.004
     burst_count: 3
-  per_issuer:
-    per_second: 0.5
-    burst_count: 5
 ```
-
 ---
 ### `rc_third_party_invite`
 
@@ -2437,14 +2405,9 @@ metrics_flags:
 ---
 ### `report_stats`
 
-Whether or not to report homeserver usage statistics. This is originally
+Whether or not to report anonymized homeserver usage statistics. This is originally
 set when generating the config. Set this option to true or false to change the current
-behavior. See
-[Reporting Homeserver Usage Statistics](../administration/monitoring/reporting_homeserver_usage_statistics.md)
-for information on what data is reported.
-
-Statistics will be reported 5 minutes after Synapse starts, and then every 3 hours
-after that.
+behavior. 
 
 Example configuration:
 ```yaml
@@ -2453,7 +2416,7 @@ report_stats: true
 ---
 ### `report_stats_endpoint`
 
-The endpoint to report homeserver usage statistics to.
+The endpoint to report the anonymized homeserver usage statistics to.
 Defaults to https://matrix.org/report-usage-stats/push
 
 Example configuration:

mypy.ini

@@ -84,6 +84,9 @@ disallow_untyped_defs = False
 [mypy-synapse.http.matrixfederationclient]
 disallow_untyped_defs = False
 
+[mypy-synapse.logging.opentracing]
+disallow_untyped_defs = False
+
 [mypy-synapse.metrics._reactor_metrics]
 disallow_untyped_defs = False
 # This module imports select.epoll. That exists on Linux, but doesn't on macOS.
@@ -141,15 +144,9 @@ ignore_missing_imports = True
 [mypy-canonicaljson]
 ignore_missing_imports = True
 
-[mypy-dictdiffer.*]
-ignore_missing_imports = True
-
 [mypy-ijson.*]
 ignore_missing_imports = True
 
-[mypy-incremental.*]
-ignore_missing_imports = True
-
 [mypy-lxml]
 ignore_missing_imports = True
 
@@ -164,9 +161,6 @@ ignore_missing_imports = True
 [mypy-parameterized.*]
 ignore_missing_imports = True
 
-[mypy-pydot.*]
-ignore_missing_imports = True
-
 [mypy-pymacaroons.*]
 ignore_missing_imports = True
 
@@ -187,3 +181,6 @@ ignore_missing_imports = True
 
 [mypy-treq.*]
 ignore_missing_imports = True
+
+[mypy-incremental.*]
+ignore_missing_imports = True

poetry.lock

@@ -228,20 +228,6 @@ wrapt = ">=1.10,<2"
 [package.extras]
 dev = ["tox", "bump2version (<1)", "sphinx (<2)", "importlib-metadata (<3)", "importlib-resources (<4)", "configparser (<5)", "sphinxcontrib-websupport (<2)", "zipp (<2)", "PyTest (<5)", "PyTest-Cov (<2.6)", "pytest", "pytest-cov"]
 
-[[package]]
-name = "dictdiffer"
-version = "0.9.0"
-description = "Dictdiffer is a library that helps you to diff and patch dictionaries."
-category = "dev"
-optional = false
-python-versions = "*"
-
-[package.extras]
-tests = ["pytest-pydocstyle (>=2.2.0)", "pytest-pycodestyle (>=2.2.0)", "pytest (>=6)", "pytest-pydocstyle (>=2)", "pytest-pycodestyle (>=2)", "pytest (==5.4.3)", "tox (>=3.7.0)", "sphinx (>=3)", "pytest-isort (>=1.2.0)", "pytest-cov (>=2.10.1)", "mock (>=1.3.0)", "check-manifest (>=0.42)"]
-numpy = ["numpy (>=1.20.0)", "numpy (>=1.18.0)", "numpy (>=1.15.0)", "numpy (>=1.13.0)"]
-docs = ["sphinx-rtd-theme (>=0.2)", "Sphinx (>=3)"]
-all = ["numpy (>=1.20.0)", "pytest-pydocstyle (>=2.2.0)", "pytest-pycodestyle (>=2.2.0)", "pytest (>=6)", "pytest-pydocstyle (>=2)", "pytest-pycodestyle (>=2)", "pytest (==5.4.3)", "numpy (>=1.18.0)", "numpy (>=1.15.0)", "numpy (>=1.13.0)", "tox (>=3.7.0)", "sphinx (>=3)", "pytest-isort (>=1.2.0)", "pytest-cov (>=2.10.1)", "mock (>=1.3.0)", "check-manifest (>=0.42)", "sphinx-rtd-theme (>=0.2)", "Sphinx (>=3)"]
-
 [[package]]
 name = "docutils"
 version = "0.18.1"
@@ -304,7 +290,7 @@ importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}
 
 [[package]]
 name = "frozendict"
-version = "2.3.0"
+version = "2.3.2"
 description = "A simple immutable dictionary"
 category = "main"
 optional = false
@@ -516,7 +502,7 @@ pyasn1 = ">=0.4.6"
 
 [[package]]
 name = "lxml"
-version = "4.8.0"
+version = "4.9.1"
 description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
 category = "main"
 optional = true
@@ -554,7 +540,7 @@ test = ["tox", "twisted", "aiounittest"]
 
 [[package]]
 name = "matrix-synapse-ldap3"
-version = "0.2.0"
+version = "0.2.1"
 description = "An LDAP3 auth provider for Synapse"
 category = "main"
 optional = true
@@ -566,7 +552,7 @@ service-identity = "*"
 Twisted = ">=15.1.0"
 
 [package.extras]
-dev = ["isort (==5.9.3)", "flake8 (==4.0.1)", "black (==21.9b0)", "types-setuptools", "mypy (==0.910)", "ldaptor", "tox", "matrix-synapse"]
+dev = ["matrix-synapse", "tox", "ldaptor", "mypy (==0.910)", "types-setuptools", "black (==22.3.0)", "flake8 (==4.0.1)", "isort (==5.9.3)"]
 
 [[package]]
 name = "mccabe"
@@ -792,17 +778,6 @@ category = "main"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
-[[package]]
-name = "pydot"
-version = "1.4.2"
-description = "Python interface to Graphviz's Dot"
-category = "dev"
-optional = false
-python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
-
-[package.dependencies]
-pyparsing = ">=2.1.4"
-
 [[package]]
 name = "pyflakes"
 version = "2.4.0"
@@ -1588,7 +1563,7 @@ url_preview = ["lxml"]
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.7.1"
-content-hash = "726ecbbacd7199e4f42517cbcafedeebf560e7fab0d45fc00f1611c9895ee4c2"
+content-hash = "e96625923122e29b6ea5964379828e321b6cede2b020fc32c6f86c09d86d1ae8"
 
 [metadata.files]
 attrs = [
@@ -1757,10 +1732,6 @@ deprecated = [
     {file = "Deprecated-1.2.13-py2.py3-none-any.whl", hash = "sha256:64756e3e14c8c5eea9795d93c524551432a0be75629f8f29e67ab8caf076c76d"},
     {file = "Deprecated-1.2.13.tar.gz", hash = "sha256:43ac5335da90c31c24ba028af536a91d41d53f9e6901ddb021bcc572ce44e38d"},
 ]
-dictdiffer = [
-    {file = "dictdiffer-0.9.0-py2.py3-none-any.whl", hash = "sha256:442bfc693cfcadaf46674575d2eba1c53b42f5e404218ca2c2ff549f2df56595"},
-    {file = "dictdiffer-0.9.0.tar.gz", hash = "sha256:17bacf5fbfe613ccf1b6d512bd766e6b21fb798822a133aa86098b8ac9997578"},
-]
 docutils = [
     {file = "docutils-0.18.1-py2.py3-none-any.whl", hash = "sha256:23010f129180089fbcd3bc08cfefccb3b890b0050e1ca00c867036e9d161b98c"},
     {file = "docutils-0.18.1.tar.gz", hash = "sha256:679987caf361a7539d76e584cbeddc311e3aee937877c87346f31debc63e9d06"},
@@ -1782,23 +1753,23 @@ flake8-comprehensions = [
     {file = "flake8_comprehensions-3.8.0-py3-none-any.whl", hash = "sha256:9406314803abe1193c064544ab14fdc43c58424c0882f6ff8a581eb73fc9bb58"},
 ]
 frozendict = [
-    {file = "frozendict-2.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e18e2abd144a9433b0a8334582843b2aa0d3b9ac8b209aaa912ad365115fe2e1"},
-    {file = "frozendict-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:96dc7a02e78da5725e5e642269bb7ae792e0c9f13f10f2e02689175ebbfedb35"},
-    {file = "frozendict-2.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:752a6dcfaf9bb20a7ecab24980e4dbe041f154509c989207caf185522ef85461"},
-    {file = "frozendict-2.3.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:5346d9fc1c936c76d33975a9a9f1a067342963105d9a403a99e787c939cc2bb2"},
-    {file = "frozendict-2.3.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:60dd2253f1bacb63a7c486ec541a968af4f985ffb06602ee8954a3d39ec6bd2e"},
-    {file = "frozendict-2.3.0-cp36-cp36m-win_amd64.whl", hash = "sha256:b2e044602ce17e5cd86724add46660fb9d80169545164e763300a3b839cb1b79"},
-    {file = "frozendict-2.3.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a27a69b1ac3591e4258325108aee62b53c0eeb6ad0a993ae68d3c7eaea980420"},
-    {file = "frozendict-2.3.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f45ef5f6b184d84744fff97b61f6b9a855e24d36b713ea2352fc723a047afa5"},
-    {file = "frozendict-2.3.0-cp37-cp37m-win_amd64.whl", hash = "sha256:2d3f5016650c0e9a192f5024e68fb4d63f670d0ee58b099ed3f5b4c62ea30ecb"},
-    {file = "frozendict-2.3.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6cf605916f50aabaaba5624c81eb270200f6c2c466c46960237a125ec8fe3ae0"},
-    {file = "frozendict-2.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f6da06e44904beae4412199d7e49be4f85c6cc168ab06b77c735ea7da5ce3454"},
-    {file = "frozendict-2.3.0-cp38-cp38-win_amd64.whl", hash = "sha256:1f34793fb409c4fa70ffd25bea87b01f3bd305fb1c6b09e7dff085b126302206"},
-    {file = "frozendict-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:fd72494a559bdcd28aa71f4aa81860269cd0b7c45fff3e2614a0a053ecfd2a13"},
-    {file = "frozendict-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00ea9166aa68cc5feed05986206fdbf35e838a09cb3feef998cf35978ff8a803"},
-    {file = "frozendict-2.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:9ffaf440648b44e0bc694c1a4701801941378ba3ba6541e17750ae4b4aeeb116"},
-    {file = "frozendict-2.3.0-py3-none-any.whl", hash = "sha256:8578fe06815fcdcc672bd5603eebc98361a5317c1c3a13b28c6c810f6ea3b323"},
-    {file = "frozendict-2.3.0.tar.gz", hash = "sha256:da4231adefc5928e7810da2732269d3ad7b5616295b3e693746392a8205ea0b5"},
+    {file = "frozendict-2.3.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4fb171d1e84d17335365877e19d17440373b47ca74a73c06f65ac0b16d01e87f"},
+    {file = "frozendict-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c0a3640e9d7533d164160b758351aa49d9e85bbe0bd76d219d4021e90ffa6a52"},
+    {file = "frozendict-2.3.2-cp310-cp310-win_amd64.whl", hash = "sha256:87cfd00fafbc147d8cd2590d1109b7db8ac8d7d5bdaa708ba46caee132b55d4d"},
+    {file = "frozendict-2.3.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:fb09761e093cfabb2f179dbfdb2521e1ec5701df714d1eb5c51fa7849027be19"},
+    {file = "frozendict-2.3.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:82176dc7adf01cf8f0193e909401939415a230a1853f4a672ec1629a06ceae18"},
+    {file = "frozendict-2.3.2-cp36-cp36m-win_amd64.whl", hash = "sha256:c1c70826aa4a50fa283fe161834ac4a3ac7c753902c980bb8b595b0998a38ddb"},
+    {file = "frozendict-2.3.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1db5035ddbed995badd1a62c4102b5e207b5aeb24472df2c60aba79639d7996b"},
+    {file = "frozendict-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4246fc4cb1413645ba4d3513939b90d979a5bae724be605a10b2b26ee12f839c"},
+    {file = "frozendict-2.3.2-cp37-cp37m-win_amd64.whl", hash = "sha256:680cd42fb0a255da1ce45678ccbd7f69da750d5243809524ebe8f45b2eda6e6b"},
+    {file = "frozendict-2.3.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6a7f3a181d6722c92a9fab12d0c5c2b006a18ca5666098531f316d1e1c8984e3"},
+    {file = "frozendict-2.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1cb866eabb3c1384a7fe88e1e1033e2b6623073589012ab637c552bf03f6364"},
+    {file = "frozendict-2.3.2-cp38-cp38-win_amd64.whl", hash = "sha256:952c5e5e664578c5c2ce8489ee0ab6a1855da02b58ef593ee728fc10d672641a"},
+    {file = "frozendict-2.3.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:608b77904cd0117cd816df605a80d0043a5326ee62529327d2136c792165a823"},
+    {file = "frozendict-2.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0eed41fd326f0bcc779837d8d9e1374da1bc9857fe3b9f2910195bbd5fff3aeb"},
+    {file = "frozendict-2.3.2-cp39-cp39-win_amd64.whl", hash = "sha256:bde28db6b5868dd3c45b3555f9d1dc5a1cca6d93591502fa5dcecce0dde6a335"},
+    {file = "frozendict-2.3.2-py3-none-any.whl", hash = "sha256:6882a9bbe08ab9b5ff96ce11bdff3fe40b114b9813bc6801261e2a7b45e20012"},
+    {file = "frozendict-2.3.2.tar.gz", hash = "sha256:7fac4542f0a13fbe704db4942f41ba3abffec5af8b100025973e59dff6a09d0d"},
 ]
 gitdb = [
     {file = "gitdb-4.0.9-py3-none-any.whl", hash = "sha256:8033ad4e853066ba6ca92050b9df2f89301b8fc8bf7e9324d412a63f8bf1a8fd"},
@@ -1966,67 +1937,76 @@ ldap3 = [
     {file = "ldap3-2.9.1.tar.gz", hash = "sha256:f3e7fc4718e3f09dda568b57100095e0ce58633bcabbed8667ce3f8fbaa4229f"},
 ]
 lxml = [
-    {file = "lxml-4.8.0-cp27-cp27m-macosx_10_14_x86_64.whl", hash = "sha256:e1ab2fac607842ac36864e358c42feb0960ae62c34aa4caaf12ada0a1fb5d99b"},
-    {file = "lxml-4.8.0-cp27-cp27m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:28d1af847786f68bec57961f31221125c29d6f52d9187c01cd34dc14e2b29430"},
-    {file = "lxml-4.8.0-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:b92d40121dcbd74831b690a75533da703750f7041b4bf951befc657c37e5695a"},
-    {file = "lxml-4.8.0-cp27-cp27m-win32.whl", hash = "sha256:e01f9531ba5420838c801c21c1b0f45dbc9607cb22ea2cf132844453bec863a5"},
-    {file = "lxml-4.8.0-cp27-cp27m-win_amd64.whl", hash = "sha256:6259b511b0f2527e6d55ad87acc1c07b3cbffc3d5e050d7e7bcfa151b8202df9"},
-    {file = "lxml-4.8.0-cp27-cp27mu-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1010042bfcac2b2dc6098260a2ed022968dbdfaf285fc65a3acf8e4eb1ffd1bc"},
-    {file = "lxml-4.8.0-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:fa56bb08b3dd8eac3a8c5b7d075c94e74f755fd9d8a04543ae8d37b1612dd170"},
-    {file = "lxml-4.8.0-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:31ba2cbc64516dcdd6c24418daa7abff989ddf3ba6d3ea6f6ce6f2ed6e754ec9"},
-    {file = "lxml-4.8.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:31499847fc5f73ee17dbe1b8e24c6dafc4e8d5b48803d17d22988976b0171f03"},
-    {file = "lxml-4.8.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:5f7d7d9afc7b293147e2d506a4596641d60181a35279ef3aa5778d0d9d9123fe"},
-    {file = "lxml-4.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:a3c5f1a719aa11866ffc530d54ad965063a8cbbecae6515acbd5f0fae8f48eaa"},
-    {file = "lxml-4.8.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6268e27873a3d191849204d00d03f65c0e343b3bcb518a6eaae05677c95621d1"},
-    {file = "lxml-4.8.0-cp310-cp310-win32.whl", hash = "sha256:330bff92c26d4aee79c5bc4d9967858bdbe73fdbdbacb5daf623a03a914fe05b"},
-    {file = "lxml-4.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:b2582b238e1658c4061ebe1b4df53c435190d22457642377fd0cb30685cdfb76"},
-    {file = "lxml-4.8.0-cp35-cp35m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a2bfc7e2a0601b475477c954bf167dee6d0f55cb167e3f3e7cefad906e7759f6"},
-    {file = "lxml-4.8.0-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:a1547ff4b8a833511eeaceacbcd17b043214fcdb385148f9c1bc5556ca9623e2"},
-    {file = "lxml-4.8.0-cp35-cp35m-win32.whl", hash = "sha256:a9f1c3489736ff8e1c7652e9dc39f80cff820f23624f23d9eab6e122ac99b150"},
-    {file = "lxml-4.8.0-cp35-cp35m-win_amd64.whl", hash = "sha256:530f278849031b0eb12f46cca0e5db01cfe5177ab13bd6878c6e739319bae654"},
-    {file = "lxml-4.8.0-cp36-cp36m-macosx_10_14_x86_64.whl", hash = "sha256:078306d19a33920004addeb5f4630781aaeabb6a8d01398045fcde085091a169"},
-    {file = "lxml-4.8.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:86545e351e879d0b72b620db6a3b96346921fa87b3d366d6c074e5a9a0b8dadb"},
-    {file = "lxml-4.8.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:24f5c5ae618395ed871b3d8ebfcbb36e3f1091fd847bf54c4de623f9107942f3"},
-    {file = "lxml-4.8.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:bbab6faf6568484707acc052f4dfc3802bdb0cafe079383fbaa23f1cdae9ecd4"},
-    {file = "lxml-4.8.0-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7993232bd4044392c47779a3c7e8889fea6883be46281d45a81451acfd704d7e"},
-    {file = "lxml-4.8.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6d6483b1229470e1d8835e52e0ff3c6973b9b97b24cd1c116dca90b57a2cc613"},
-    {file = "lxml-4.8.0-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:ad4332a532e2d5acb231a2e5d33f943750091ee435daffca3fec0a53224e7e33"},
-    {file = "lxml-4.8.0-cp36-cp36m-win32.whl", hash = "sha256:db3535733f59e5605a88a706824dfcb9bd06725e709ecb017e165fc1d6e7d429"},
-    {file = "lxml-4.8.0-cp36-cp36m-win_amd64.whl", hash = "sha256:5f148b0c6133fb928503cfcdfdba395010f997aa44bcf6474fcdd0c5398d9b63"},
-    {file = "lxml-4.8.0-cp37-cp37m-macosx_10_14_x86_64.whl", hash = "sha256:8a31f24e2a0b6317f33aafbb2f0895c0bce772980ae60c2c640d82caac49628a"},
-    {file = "lxml-4.8.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:719544565c2937c21a6f76d520e6e52b726d132815adb3447ccffbe9f44203c4"},
-    {file = "lxml-4.8.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:c0b88ed1ae66777a798dc54f627e32d3b81c8009967c63993c450ee4cbcbec15"},
-    {file = "lxml-4.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:fa9b7c450be85bfc6cd39f6df8c5b8cbd76b5d6fc1f69efec80203f9894b885f"},
-    {file = "lxml-4.8.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e9f84ed9f4d50b74fbc77298ee5c870f67cb7e91dcdc1a6915cb1ff6a317476c"},
-    {file = "lxml-4.8.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:1d650812b52d98679ed6c6b3b55cbb8fe5a5460a0aef29aeb08dc0b44577df85"},
-    {file = "lxml-4.8.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:80bbaddf2baab7e6de4bc47405e34948e694a9efe0861c61cdc23aa774fcb141"},
-    {file = "lxml-4.8.0-cp37-cp37m-win32.whl", hash = "sha256:6f7b82934c08e28a2d537d870293236b1000d94d0b4583825ab9649aef7ddf63"},
-    {file = "lxml-4.8.0-cp37-cp37m-win_amd64.whl", hash = "sha256:e1fd7d2fe11f1cb63d3336d147c852f6d07de0d0020d704c6031b46a30b02ca8"},
-    {file = "lxml-4.8.0-cp38-cp38-macosx_10_14_x86_64.whl", hash = "sha256:5045ee1ccd45a89c4daec1160217d363fcd23811e26734688007c26f28c9e9e7"},
-    {file = "lxml-4.8.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:0c1978ff1fd81ed9dcbba4f91cf09faf1f8082c9d72eb122e92294716c605428"},
-    {file = "lxml-4.8.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:52cbf2ff155b19dc4d4100f7442f6a697938bf4493f8d3b0c51d45568d5666b5"},
-    {file = "lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:ce13d6291a5f47c1c8dbd375baa78551053bc6b5e5c0e9bb8e39c0a8359fd52f"},
-    {file = "lxml-4.8.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e11527dc23d5ef44d76fef11213215c34f36af1608074561fcc561d983aeb870"},
-    {file = "lxml-4.8.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:60d2f60bd5a2a979df28ab309352cdcf8181bda0cca4529769a945f09aba06f9"},
-    {file = "lxml-4.8.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:62f93eac69ec0f4be98d1b96f4d6b964855b8255c345c17ff12c20b93f247b68"},
-    {file = "lxml-4.8.0-cp38-cp38-win32.whl", hash = "sha256:20b8a746a026017acf07da39fdb10aa80ad9877046c9182442bf80c84a1c4696"},
-    {file = "lxml-4.8.0-cp38-cp38-win_amd64.whl", hash = "sha256:891dc8f522d7059ff0024cd3ae79fd224752676447f9c678f2a5c14b84d9a939"},
-    {file = "lxml-4.8.0-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:b6fc2e2fb6f532cf48b5fed57567ef286addcef38c28874458a41b7837a57807"},
-    {file = "lxml-4.8.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:74eb65ec61e3c7c019d7169387d1b6ffcfea1b9ec5894d116a9a903636e4a0b1"},
-    {file = "lxml-4.8.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:627e79894770783c129cc5e89b947e52aa26e8e0557c7e205368a809da4b7939"},
-    {file = "lxml-4.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:545bd39c9481f2e3f2727c78c169425efbfb3fbba6e7db4f46a80ebb249819ca"},
-    {file = "lxml-4.8.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5a58d0b12f5053e270510bf12f753a76aaf3d74c453c00942ed7d2c804ca845c"},
-    {file = "lxml-4.8.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:ec4b4e75fc68da9dc0ed73dcdb431c25c57775383fec325d23a770a64e7ebc87"},
-    {file = "lxml-4.8.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:5804e04feb4e61babf3911c2a974a5b86f66ee227cc5006230b00ac6d285b3a9"},
-    {file = "lxml-4.8.0-cp39-cp39-win32.whl", hash = "sha256:aa0cf4922da7a3c905d000b35065df6184c0dc1d866dd3b86fd961905bbad2ea"},
-    {file = "lxml-4.8.0-cp39-cp39-win_amd64.whl", hash = "sha256:dd10383f1d6b7edf247d0960a3db274c07e96cf3a3fc7c41c8448f93eac3fb1c"},
-    {file = "lxml-4.8.0-pp37-pypy37_pp73-macosx_10_14_x86_64.whl", hash = "sha256:2403a6d6fb61c285969b71f4a3527873fe93fd0abe0832d858a17fe68c8fa507"},
-    {file = "lxml-4.8.0-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:986b7a96228c9b4942ec420eff37556c5777bfba6758edcb95421e4a614b57f9"},
-    {file = "lxml-4.8.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:6fe4ef4402df0250b75ba876c3795510d782def5c1e63890bde02d622570d39e"},
-    {file = "lxml-4.8.0-pp38-pypy38_pp73-macosx_10_14_x86_64.whl", hash = "sha256:f10ce66fcdeb3543df51d423ede7e238be98412232fca5daec3e54bcd16b8da0"},
-    {file = "lxml-4.8.0-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:730766072fd5dcb219dd2b95c4c49752a54f00157f322bc6d71f7d2a31fecd79"},
-    {file = "lxml-4.8.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:8b99ec73073b37f9ebe8caf399001848fced9c08064effdbfc4da2b5a8d07b93"},
-    {file = "lxml-4.8.0.tar.gz", hash = "sha256:f63f62fc60e6228a4ca9abae28228f35e1bd3ce675013d1dfb828688d50c6e23"},
+    {file = "lxml-4.9.1-cp27-cp27m-macosx_10_15_x86_64.whl", hash = "sha256:98cafc618614d72b02185ac583c6f7796202062c41d2eeecdf07820bad3295ed"},
+    {file = "lxml-4.9.1-cp27-cp27m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c62e8dd9754b7debda0c5ba59d34509c4688f853588d75b53c3791983faa96fc"},
+    {file = "lxml-4.9.1-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:21fb3d24ab430fc538a96e9fbb9b150029914805d551deeac7d7822f64631dfc"},
+    {file = "lxml-4.9.1-cp27-cp27m-win32.whl", hash = "sha256:86e92728ef3fc842c50a5cb1d5ba2bc66db7da08a7af53fb3da79e202d1b2cd3"},
+    {file = "lxml-4.9.1-cp27-cp27m-win_amd64.whl", hash = "sha256:4cfbe42c686f33944e12f45a27d25a492cc0e43e1dc1da5d6a87cbcaf2e95627"},
+    {file = "lxml-4.9.1-cp27-cp27mu-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:dad7b164905d3e534883281c050180afcf1e230c3d4a54e8038aa5cfcf312b84"},
+    {file = "lxml-4.9.1-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:a614e4afed58c14254e67862456d212c4dcceebab2eaa44d627c2ca04bf86837"},
+    {file = "lxml-4.9.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:f9ced82717c7ec65a67667bb05865ffe38af0e835cdd78728f1209c8fffe0cad"},
+    {file = "lxml-4.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:d9fc0bf3ff86c17348dfc5d322f627d78273eba545db865c3cd14b3f19e57fa5"},
+    {file = "lxml-4.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:e5f66bdf0976ec667fc4594d2812a00b07ed14d1b44259d19a41ae3fff99f2b8"},
+    {file = "lxml-4.9.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:fe17d10b97fdf58155f858606bddb4e037b805a60ae023c009f760d8361a4eb8"},
+    {file = "lxml-4.9.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8caf4d16b31961e964c62194ea3e26a0e9561cdf72eecb1781458b67ec83423d"},
+    {file = "lxml-4.9.1-cp310-cp310-win32.whl", hash = "sha256:4780677767dd52b99f0af1f123bc2c22873d30b474aa0e2fc3fe5e02217687c7"},
+    {file = "lxml-4.9.1-cp310-cp310-win_amd64.whl", hash = "sha256:b122a188cd292c4d2fcd78d04f863b789ef43aa129b233d7c9004de08693728b"},
+    {file = "lxml-4.9.1-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:be9eb06489bc975c38706902cbc6888f39e946b81383abc2838d186f0e8b6a9d"},
+    {file = "lxml-4.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:f1be258c4d3dc609e654a1dc59d37b17d7fef05df912c01fc2e15eb43a9735f3"},
+    {file = "lxml-4.9.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:927a9dd016d6033bc12e0bf5dee1dde140235fc8d0d51099353c76081c03dc29"},
+    {file = "lxml-4.9.1-cp35-cp35m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:9232b09f5efee6a495a99ae6824881940d6447debe272ea400c02e3b68aad85d"},
+    {file = "lxml-4.9.1-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:04da965dfebb5dac2619cb90fcf93efdb35b3c6994fea58a157a834f2f94b318"},
+    {file = "lxml-4.9.1-cp35-cp35m-win32.whl", hash = "sha256:4d5bae0a37af799207140652a700f21a85946f107a199bcb06720b13a4f1f0b7"},
+    {file = "lxml-4.9.1-cp35-cp35m-win_amd64.whl", hash = "sha256:4878e667ebabe9b65e785ac8da4d48886fe81193a84bbe49f12acff8f7a383a4"},
+    {file = "lxml-4.9.1-cp36-cp36m-macosx_10_15_x86_64.whl", hash = "sha256:1355755b62c28950f9ce123c7a41460ed9743c699905cbe664a5bcc5c9c7c7fb"},
+    {file = "lxml-4.9.1-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:bcaa1c495ce623966d9fc8a187da80082334236a2a1c7e141763ffaf7a405067"},
+    {file = "lxml-4.9.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6eafc048ea3f1b3c136c71a86db393be36b5b3d9c87b1c25204e7d397cee9536"},
+    {file = "lxml-4.9.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:13c90064b224e10c14dcdf8086688d3f0e612db53766e7478d7754703295c7c8"},
+    {file = "lxml-4.9.1-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:206a51077773c6c5d2ce1991327cda719063a47adc02bd703c56a662cdb6c58b"},
+    {file = "lxml-4.9.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:e8f0c9d65da595cfe91713bc1222af9ecabd37971762cb830dea2fc3b3bb2acf"},
+    {file = "lxml-4.9.1-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:8f0a4d179c9a941eb80c3a63cdb495e539e064f8054230844dcf2fcb812b71d3"},
+    {file = "lxml-4.9.1-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:830c88747dce8a3e7525defa68afd742b4580df6aa2fdd6f0855481e3994d391"},
+    {file = "lxml-4.9.1-cp36-cp36m-win32.whl", hash = "sha256:1e1cf47774373777936c5aabad489fef7b1c087dcd1f426b621fda9dcc12994e"},
+    {file = "lxml-4.9.1-cp36-cp36m-win_amd64.whl", hash = "sha256:5974895115737a74a00b321e339b9c3f45c20275d226398ae79ac008d908bff7"},
+    {file = "lxml-4.9.1-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:1423631e3d51008871299525b541413c9b6c6423593e89f9c4cfbe8460afc0a2"},
+    {file = "lxml-4.9.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:2aaf6a0a6465d39b5ca69688fce82d20088c1838534982996ec46633dc7ad6cc"},
+    {file = "lxml-4.9.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:9f36de4cd0c262dd9927886cc2305aa3f2210db437aa4fed3fb4940b8bf4592c"},
+    {file = "lxml-4.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:ae06c1e4bc60ee076292e582a7512f304abdf6c70db59b56745cca1684f875a4"},
+    {file = "lxml-4.9.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:57e4d637258703d14171b54203fd6822fda218c6c2658a7d30816b10995f29f3"},
+    {file = "lxml-4.9.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6d279033bf614953c3fc4a0aa9ac33a21e8044ca72d4fa8b9273fe75359d5cca"},
+    {file = "lxml-4.9.1-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:a60f90bba4c37962cbf210f0188ecca87daafdf60271f4c6948606e4dabf8785"},
+    {file = "lxml-4.9.1-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:6ca2264f341dd81e41f3fffecec6e446aa2121e0b8d026fb5130e02de1402785"},
+    {file = "lxml-4.9.1-cp37-cp37m-win32.whl", hash = "sha256:27e590352c76156f50f538dbcebd1925317a0f70540f7dc8c97d2931c595783a"},
+    {file = "lxml-4.9.1-cp37-cp37m-win_amd64.whl", hash = "sha256:eea5d6443b093e1545ad0210e6cf27f920482bfcf5c77cdc8596aec73523bb7e"},
+    {file = "lxml-4.9.1-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:f05251bbc2145349b8d0b77c0d4e5f3b228418807b1ee27cefb11f69ed3d233b"},
+    {file = "lxml-4.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:487c8e61d7acc50b8be82bda8c8d21d20e133c3cbf41bd8ad7eb1aaeb3f07c97"},
+    {file = "lxml-4.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:8d1a92d8e90b286d491e5626af53afef2ba04da33e82e30744795c71880eaa21"},
+    {file = "lxml-4.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:b570da8cd0012f4af9fa76a5635cd31f707473e65a5a335b186069d5c7121ff2"},
+    {file = "lxml-4.9.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5ef87fca280fb15342726bd5f980f6faf8b84a5287fcc2d4962ea8af88b35130"},
+    {file = "lxml-4.9.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:93e414e3206779ef41e5ff2448067213febf260ba747fc65389a3ddaa3fb8715"},
+    {file = "lxml-4.9.1-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:6653071f4f9bac46fbc30f3c7838b0e9063ee335908c5d61fb7a4a86c8fd2036"},
+    {file = "lxml-4.9.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:32a73c53783becdb7eaf75a2a1525ea8e49379fb7248c3eeefb9412123536387"},
+    {file = "lxml-4.9.1-cp38-cp38-win32.whl", hash = "sha256:1a7c59c6ffd6ef5db362b798f350e24ab2cfa5700d53ac6681918f314a4d3b94"},
+    {file = "lxml-4.9.1-cp38-cp38-win_amd64.whl", hash = "sha256:1436cf0063bba7888e43f1ba8d58824f085410ea2025befe81150aceb123e345"},
+    {file = "lxml-4.9.1-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:4beea0f31491bc086991b97517b9683e5cfb369205dac0148ef685ac12a20a67"},
+    {file = "lxml-4.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:41fb58868b816c202e8881fd0f179a4644ce6e7cbbb248ef0283a34b73ec73bb"},
+    {file = "lxml-4.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:bd34f6d1810d9354dc7e35158aa6cc33456be7706df4420819af6ed966e85448"},
+    {file = "lxml-4.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:edffbe3c510d8f4bf8640e02ca019e48a9b72357318383ca60e3330c23aaffc7"},
+    {file = "lxml-4.9.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6d949f53ad4fc7cf02c44d6678e7ff05ec5f5552b235b9e136bd52e9bf730b91"},
+    {file = "lxml-4.9.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:079b68f197c796e42aa80b1f739f058dcee796dc725cc9a1be0cdb08fc45b000"},
+    {file = "lxml-4.9.1-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:9c3a88d20e4fe4a2a4a84bf439a5ac9c9aba400b85244c63a1ab7088f85d9d25"},
+    {file = "lxml-4.9.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:4e285b5f2bf321fc0857b491b5028c5f276ec0c873b985d58d7748ece1d770dd"},
+    {file = "lxml-4.9.1-cp39-cp39-win32.whl", hash = "sha256:ef72013e20dd5ba86a8ae1aed7f56f31d3374189aa8b433e7b12ad182c0d2dfb"},
+    {file = "lxml-4.9.1-cp39-cp39-win_amd64.whl", hash = "sha256:10d2017f9150248563bb579cd0d07c61c58da85c922b780060dcc9a3aa9f432d"},
+    {file = "lxml-4.9.1-pp37-pypy37_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0538747a9d7827ce3e16a8fdd201a99e661c7dee3c96c885d8ecba3c35d1032c"},
+    {file = "lxml-4.9.1-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:0645e934e940107e2fdbe7c5b6fb8ec6232444260752598bc4d09511bd056c0b"},
+    {file = "lxml-4.9.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:6daa662aba22ef3258934105be2dd9afa5bb45748f4f702a3b39a5bf53a1f4dc"},
+    {file = "lxml-4.9.1-pp38-pypy38_pp73-macosx_10_15_x86_64.whl", hash = "sha256:603a464c2e67d8a546ddaa206d98e3246e5db05594b97db844c2f0a1af37cf5b"},
+    {file = "lxml-4.9.1-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:c4b2e0559b68455c085fb0f6178e9752c4be3bba104d6e881eb5573b399d1eb2"},
+    {file = "lxml-4.9.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:0f3f0059891d3254c7b5fb935330d6db38d6519ecd238ca4fce93c234b4a0f73"},
+    {file = "lxml-4.9.1-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:c852b1530083a620cb0de5f3cd6826f19862bafeaf77586f1aef326e49d95f0c"},
+    {file = "lxml-4.9.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:287605bede6bd36e930577c5925fcea17cb30453d96a7b4c63c14a257118dbb9"},
+    {file = "lxml-4.9.1.tar.gz", hash = "sha256:fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f"},
 ]
 markupsafe = [
     {file = "MarkupSafe-2.1.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:3028252424c72b2602a323f70fbf50aa80a5d3aa616ea6add4ba21ae9cc9da4c"},
@@ -2075,8 +2055,8 @@ matrix-common = [
     {file = "matrix_common-1.2.1.tar.gz", hash = "sha256:a99dcf02a6bd95b24a5a61b354888a2ac92bf2b4b839c727b8dd9da2cdfa3853"},
 ]
 matrix-synapse-ldap3 = [
-    {file = "matrix-synapse-ldap3-0.2.0.tar.gz", hash = "sha256:91a0715b43a41ec3033244174fca20846836da98fda711fb01687f7199eecd2e"},
-    {file = "matrix_synapse_ldap3-0.2.0-py3-none-any.whl", hash = "sha256:0128ca7c3058987adc2e8a88463bb46879915bfd3d373309632813b353e30f9f"},
+    {file = "matrix-synapse-ldap3-0.2.1.tar.gz", hash = "sha256:bfb4390f4a262ffb0d6f057ff3aeb1e46d4e52ff420a064d795fb4f555f00285"},
+    {file = "matrix_synapse_ldap3-0.2.1-py3-none-any.whl", hash = "sha256:1b3310a60f1d06466f35905a269b6df95747fd1305f2b7fe638f373963b2aa2c"},
 ]
 mccabe = [
     {file = "mccabe-0.6.1-py2.py3-none-any.whl", hash = "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42"},
@@ -2280,10 +2260,6 @@ pycparser = [
     {file = "pycparser-2.21-py2.py3-none-any.whl", hash = "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9"},
     {file = "pycparser-2.21.tar.gz", hash = "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"},
 ]
-pydot = [
-    {file = "pydot-1.4.2-py2.py3-none-any.whl", hash = "sha256:66c98190c65b8d2e2382a441b4c0edfdb4f4c025ef9cb9874de478fb0793a451"},
-    {file = "pydot-1.4.2.tar.gz", hash = "sha256:248081a39bcb56784deb018977e428605c1c758f10897a339fce1dd728ff007d"},
-]
 pyflakes = [
     {file = "pyflakes-2.4.0-py2.py3-none-any.whl", hash = "sha256:3bb3a3f256f4b7968c9c788781e4ff07dce46bdf12339dcda61053375426ee2e"},
     {file = "pyflakes-2.4.0.tar.gz", hash = "sha256:05a85c2872edf37a4ed30b0cce2f6093e1d0581f8c19d7393122da7e25b2b24c"},

pyproject.toml

@@ -54,7 +54,7 @@ skip_gitignore = true
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.63.1"
+version = "1.63.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -110,9 +110,7 @@ jsonschema = ">=3.0.0"
 frozendict = ">=1,!=2.1.2"
 # We require 2.1.0 or higher for type hints. Previous guard was >= 1.1.0
 unpaddedbase64 = ">=2.1.0"
-# We require 1.5.0 to work around an issue when running against the C implementation of
-# frozendict: https://github.com/matrix-org/python-canonicaljson/issues/36
-canonicaljson = "^1.5.0"
+canonicaljson = "^1.4.0"
 # we use the type definitions added in signedjson 1.1.
 signedjson = "^1.1.0"
 # validating SSL certs for IP addresses requires service_identity 18.1.
@@ -281,10 +279,6 @@ twine = "*"
 # Towncrier min version comes from #3425. Rationale unclear.
 towncrier = ">=18.6.0rc1"
 
-# Used by the debug_state_res dev script
-dictdiffer = ">=0.9.0"
-pydot = ">=1.4.2"
-
 [build-system]
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"

scripts-dev/build_debian_packages.py

@@ -26,6 +26,7 @@ DISTS = (
     "debian:bookworm",
     "debian:sid",
     "ubuntu:focal",  # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
+    "ubuntu:impish",  # 21.10  (EOL 2022-07)
     "ubuntu:jammy",  # 22.04 LTS (EOL 2027-04)
 )
 

scripts-dev/debug_state_res.py

@@ -1,332 +0,0 @@
-#! /usr/bin/env python
-import argparse
-import logging
-import sys
-from pprint import pformat
-from typing import Awaitable, Callable, Collection, Dict, List, Optional, Tuple, cast
-from unittest.mock import MagicMock, patch
-
-import dictdiffer
-import pydot
-import yaml
-
-from twisted.internet import task
-
-from synapse.config._base import RootConfig
-from synapse.config.cache import CacheConfig
-from synapse.config.database import DatabaseConfig
-from synapse.config.workers import WorkerConfig
-from synapse.events import EventBase
-from synapse.server import HomeServer
-from synapse.state import StateResolutionStore
-from synapse.storage.databases.main.event_federation import EventFederationWorkerStore
-from synapse.storage.databases.main.events_worker import EventsWorkerStore
-from synapse.storage.databases.main.room import RoomWorkerStore
-from synapse.storage.databases.main.state import StateGroupWorkerStore
-from synapse.storage.state import StateFilter
-from synapse.types import ISynapseReactor, StateMap
-
-"""This monstrosity is useful for visualising and debugging state resolution problems.
-
-
-"""
-
-logger = logging.getLogger(sys.argv[0])
-
-
-# Bits of the HomeServer Machinery we need to talk to the DB.
-class Config(RootConfig):
-    config_classes = [DatabaseConfig, WorkerConfig, CacheConfig]
-
-
-def load_config(source: str) -> Config:
-    data = yaml.safe_load(source)
-    data["worker_name"] = "stateres-debug"
-
-    config = Config()
-    config.parse_config_dict(data, "DUMMYPATH", "DUMMYPATH")
-    config.key = MagicMock()  # Don't bother creating signing keys
-    return config
-
-
-class DataStore(
-    StateGroupWorkerStore,
-    EventFederationWorkerStore,
-    EventsWorkerStore,
-    RoomWorkerStore,
-):
-    pass
-
-
-class MockHomeserver(HomeServer):
-    DATASTORE_CLASS = DataStore  # type: ignore [assignment]
-
-    def __init__(self, config: Config):
-        super(MockHomeserver, self).__init__(
-            hostname="stateres-debug",
-            config=config,  # type: ignore[arg-type]
-        )
-
-
-# Functions for drawing graphviz diagrams via `pydot`.
-def node(
-    event: EventBase, suffix: Optional[str] = None, **kwargs: object
-) -> pydot.Node:
-    if "label" not in kwargs:
-        label = (
-            f"{event.event_id}\n{event.sender}: {(event.type,event.get_state_key())}"
-        )
-        if event.type == "m.room.member":
-            label += f" ({event.membership.upper()})"
-        if suffix:
-            label += f"\n{suffix}"
-        kwargs["label"] = label
-    type_to_shape: Dict[str, str] = {}  # {"m.room.member": "oval"}
-    if event.type in type_to_shape:
-        kwargs.setdefault("shape", type_to_shape[event.type])
-
-    q = pydot.quote_if_necessary
-    return pydot.Node(q(event.event_id), **kwargs)
-
-
-def edge(source: EventBase, target: EventBase, **kwargs: object) -> pydot.Edge:
-    return pydot.Edge(
-        pydot.quote_if_necessary(source.event_id),
-        pydot.quote_if_necessary(target.event_id),
-        **kwargs,
-    )
-
-
-async def dump_mainlines(
-    hs: MockHomeserver,
-    resolve_point: Optional[EventBase],
-    events: Collection[EventBase],
-    extras: Collection[str],
-    watch_func: Optional[Callable[[EventBase], Awaitable[str]]] = None,
-) -> None:
-    """Visualise the auth DAG above a given `starting_event`.
-
-    Starting with the given event's parents and any `extras` of interest, we search in
-    their auth events for power levels, join rules and sender membership events.
-    We recursively repeat this process for any events found during the search
-    until we have no more auth-ancestors of interest to find.
-
-    In this way we build up a subset of the auth chain of the `starting_event`.
-    (In particular we omit edges to m.room.create: they are everywhere and convey no
-    information.)
-
-    An optional `watch_func` allows us to annotate the events we see with a string of
-    our choice. This can be useful if we want to track a single piece of state through
-    the auth DAG.
-    """
-    graph = pydot.Dot(rankdir="BT")
-    graph.set_node_defaults(shape="box", style="filled")
-
-    async def new_node(event: EventBase, **kwargs: object) -> pydot.Node:
-        suffix = await watch_func(event) if watch_func else None
-        return node(event, suffix, **kwargs)
-
-    seen = set()
-    todo: List[EventBase] = []
-
-    if resolve_point:
-        graph.add_node(await new_node(resolve_point, fillcolor="#6699cc"))
-        seen.add(resolve_point.event_id)
-
-    for parent in events:
-        graph.add_node(await new_node(parent, fillcolor="#6699cc"))
-        seen.add(parent.event_id)
-        todo.append(parent)
-        if resolve_point:
-            graph.add_edge(edge(resolve_point, parent, style="dashed"))
-
-    if extras:
-        logger.debug(extras)
-        extra_events = await hs.get_datastores().main.get_events(extras)
-        logger.debug(extra_events)
-        for extra_event in extra_events.values():
-            if extra_event.event_id in seen:
-                continue
-            graph.add_node(await new_node(extra_event, fillcolor="#6699ee"))
-            todo.append(extra_event)
-
-    async def fetch_auth_events(event: EventBase) -> StateMap[EventBase]:
-        return {
-            (e.type, e.state_key): e
-            for e in (
-                await hs.get_datastores().main.get_events(event.auth_event_ids())
-            ).values()
-        }
-
-    while todo:
-        event = todo.pop()
-        auth_events = await fetch_auth_events(event)
-
-        for key, edge_style in [
-            (("m.room.power_levels", ""), "solid"),
-            (("m.room.join_rules", ""), "solid"),
-            (("m.room.member", event.sender), "dotted"),
-            # TODO: handle that state_key might be missing
-            # (("m.room.member", event.state_key), "solid"),
-        ]:
-            auth_event = auth_events.get(key)
-            if auth_event:
-                if auth_event.event_id not in seen:
-                    node_options = {}
-                    if key[0] == "m.room.power_levels":
-                        node_options["fillcolor"] = "#ffcccc"
-                    elif key[0] == "m.room.join_rules":
-                        node_options["fillcolor"] = "#cc9966"
-                    elif key == ("m.room.member", event.sender):
-                        auth_events_2 = await fetch_auth_events(auth_event)
-                        if ("m.room.member", event.sender) not in auth_events_2:
-                            # auth_event is the first join of that sender
-                            node_options["fillcolor"] = "#33ff33"
-                        else:
-                            node_options["fillcolor"] = "#ccffcc"
-
-                    graph.add_node(await new_node(auth_event, **node_options))
-                    seen.add(auth_event.event_id)
-                    todo.append(auth_event)
-                graph.add_edge(edge(event, auth_event, style=edge_style))
-
-    # TODO: make this location configurable
-    graph.write_svg("mainlines.svg")
-
-
-# The main logic and the arguments we need to invoke it.
-parser = argparse.ArgumentParser(
-    description="Debug the stateres calculation of a specific event."
-)
-parser.add_argument(
-    "config_file", help="Synapse config file", type=argparse.FileType("r")
-)
-parser.add_argument("--verbose", "-v", help="Log verbosely", action="store_true")
-parser.add_argument("-d", "--draw", help="Render auth DAG", action="store_true")
-parser.add_argument(
-    "event_ids",
-    help="""\
-The event ID(s) to be resolved.\
-
-If a single event is given, resolve across all of its parents to compute the state
-before the given event. If multiple events are given, resolve across them directly.
-""",
-    nargs="+",
-)
-parser.add_argument(
-    "-e",
-    "--extra",
-    dest="extras",
-    help=(
-        "An extra event to include in the auth DAG when using the `--draw` flag. "
-        "Can be provided multiple times."
-    ),
-    action="append",
-)
-parser.add_argument(
-    "--watch",
-    help="Track a piece of state in the auth DAG when using the `--draw` flag.",
-    default=None,
-    nargs=2,
-    metavar=("TYPE", "STATE_KEY"),
-)
-
-
-async def debug_specific_stateres(
-    reactor: ISynapseReactor, hs: MockHomeserver, args: argparse.Namespace
-) -> None:
-    """Recompute the state at the given event.
-
-    This produces
-    - a file called `mainline.svg` representing the auth chain of the given event,
-    - logging from state resolution calculations, written to stdout,
-    - the recomputed and stored state, written to stdout, and
-    - their difference, written to stdout.
-    """
-    DEBUG_AT_EVENT = len(args.event_ids) == 1
-
-    if DEBUG_AT_EVENT:
-        resolve_point = await hs.get_datastores().main.get_event(args.event_ids[0])
-        prev_event_ids = resolve_point.prev_event_ids()
-    else:
-        resolve_point = None
-        prev_event_ids = args.event_ids
-
-    parent_events = (await hs.get_datastores().main.get_events(prev_event_ids)).values()
-    sample_event = next(iter(parent_events))
-
-    logger.info("Resolving across %d parents, %s", len(prev_event_ids), prev_event_ids)
-    state_after_parents = [
-        await hs.get_storage_controllers().state.get_state_ids_for_event(prev_event_id)
-        for prev_event_id in prev_event_ids
-    ]
-
-    if args.watch is not None:
-        key_pair = cast(Tuple[str, str], tuple(args.watch))
-        filter = StateFilter.from_types([key_pair])
-
-        watch_func: Optional[Callable[[EventBase], Awaitable[str]]]
-
-        async def watch_func(event: EventBase) -> str:
-            try:
-                result = (
-                    await hs.get_storage_controllers().state.get_state_ids_for_event(
-                        event.event_id, filter
-                    )
-                )
-            except RuntimeError:
-                return f"\n{key_pair}: <Event unavailable :(>"
-            else:
-                return f"\n{key_pair}: {result.get(key_pair, '<No event in state>')}"
-
-    else:
-        watch_func = None
-
-    if args.draw:
-        await dump_mainlines(hs, resolve_point, parent_events, args.extras, watch_func)
-
-    result = await hs.get_state_resolution_handler().resolve_events_with_store(
-        sample_event.room_id,
-        sample_event.room_version.identifier,
-        state_after_parents,
-        event_map=None,
-        state_res_store=StateResolutionStore(hs.get_datastores().main),
-    )
-
-    logger.info("State resolved:")
-    logger.info(pformat(result))
-
-    if DEBUG_AT_EVENT:
-        logger.info("Stored state at %s:", sample_event.event_id)
-        stored_state = await hs.get_storage_controllers().state.get_state_ids_for_event(
-            sample_event.event_id
-        )
-        logger.info(pformat(stored_state))
-
-        # TODO make this a like-for-like comparison.
-        logger.info("Diff from stored (after event) to resolved (before event):")
-        for change in dictdiffer.diff(stored_state, result):
-            logger.info(pformat(change))
-
-
-# Entrypoint.
-if __name__ == "__main__":
-    args = parser.parse_args()
-    logging.basicConfig(
-        format="%(asctime)s %(name)s:%(lineno)d %(levelname)s %(message)s",
-        level=logging.DEBUG if args.verbose else logging.INFO,
-        stream=sys.stdout,
-    )
-    # Suppress logs we aren't interested in.
-    logging.getLogger("synapse.util").setLevel(logging.ERROR)
-    logging.getLogger("synapse.storage").setLevel(logging.ERROR)
-
-    config = load_config(args.config_file)
-    hs = MockHomeserver(config)
-    # Patch out enough stuff so we can work with a readonly DB connection.
-    with patch("synapse.storage.databases.prepare_database"), patch(
-        "synapse.storage.database.BackgroundUpdater"
-    ), patch("synapse.storage.databases.main.events_worker.MultiWriterIdGenerator"):
-        hs.setup()
-
-    task.react(debug_specific_stateres, [hs, parser.parse_args()])

synapse/_scripts/generate_config.py

@@ -33,7 +33,7 @@ def main() -> None:
     parser.add_argument(
         "--report-stats",
         action="store",
-        help="Whether the generated config reports homeserver usage statistics",
+        help="Whether the generated config reports anonymized usage statistics",
         choices=["yes", "no"],
     )
 

synapse/app/admin_cmd.py

@@ -28,22 +28,19 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.events import EventBase
 from synapse.handlers.admin import ExfiltrationWriter
+from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
+from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
+from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
+from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
+from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
-from synapse.storage.databases.main.account_data import AccountDataWorkerStore
-from synapse.storage.databases.main.appservice import (
-    ApplicationServiceTransactionWorkerStore,
-    ApplicationServiceWorkerStore,
-)
-from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
-from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
-from synapse.storage.databases.main.registration import RegistrationWorkerStore
 from synapse.storage.databases.main.room import RoomWorkerStore
-from synapse.storage.databases.main.tags import TagsWorkerStore
 from synapse.types import StateMap
 from synapse.util import SYNAPSE_VERSION
 from synapse.util.logcontext import LoggingContext
@@ -52,17 +49,16 @@ logger = logging.getLogger("synapse.app.admin_cmd")
 
 
 class AdminCmdSlavedStore(
+    SlavedReceiptsStore,
+    SlavedAccountDataStore,
+    SlavedApplicationServiceStore,
+    SlavedRegistrationStore,
     SlavedFilteringStore,
+    SlavedDeviceInboxStore,
     SlavedDeviceStore,
     SlavedPushRuleStore,
     SlavedEventStore,
-    TagsWorkerStore,
-    DeviceInboxWorkerStore,
-    AccountDataWorkerStore,
-    ApplicationServiceTransactionWorkerStore,
-    ApplicationServiceWorkerStore,
-    RegistrationWorkerStore,
-    ReceiptsWorkerStore,
+    BaseSlavedStore,
     RoomWorkerStore,
 ):
     def __init__(

synapse/app/generic_worker.py

@@ -48,12 +48,20 @@ from synapse.http.site import SynapseRequest, SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
+from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
+from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
+from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
+from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
 from synapse.replication.slave.storage.keys import SlavedKeyStore
+from synapse.replication.slave.storage.profile import SlavedProfileStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
+from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
+from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.rest.admin import register_servlets_for_media_repo
 from synapse.rest.client import (
     account_data,
@@ -92,15 +100,8 @@ from synapse.rest.key.v2 import KeyApiV2Resource
 from synapse.rest.synapse.client import build_synapse_client_resource_tree
 from synapse.rest.well_known import well_known_resource
 from synapse.server import HomeServer
-from synapse.storage.databases.main.account_data import AccountDataWorkerStore
-from synapse.storage.databases.main.appservice import (
-    ApplicationServiceTransactionWorkerStore,
-    ApplicationServiceWorkerStore,
-)
 from synapse.storage.databases.main.censor_events import CensorEventsStore
 from synapse.storage.databases.main.client_ips import ClientIpWorkerStore
-from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
-from synapse.storage.databases.main.directory import DirectoryWorkerStore
 from synapse.storage.databases.main.e2e_room_keys import EndToEndRoomKeyStore
 from synapse.storage.databases.main.lock import LockStore
 from synapse.storage.databases.main.media_repository import MediaRepositoryStore
@@ -109,15 +110,11 @@ from synapse.storage.databases.main.monthly_active_users import (
     MonthlyActiveUsersWorkerStore,
 )
 from synapse.storage.databases.main.presence import PresenceStore
-from synapse.storage.databases.main.profile import ProfileWorkerStore
-from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
-from synapse.storage.databases.main.registration import RegistrationWorkerStore
 from synapse.storage.databases.main.room import RoomWorkerStore
 from synapse.storage.databases.main.room_batch import RoomBatchStore
 from synapse.storage.databases.main.search import SearchStore
 from synapse.storage.databases.main.session import SessionStore
 from synapse.storage.databases.main.stats import StatsStore
-from synapse.storage.databases.main.tags import TagsWorkerStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
@@ -230,11 +227,11 @@ class GenericWorkerSlavedStore(
     UIAuthWorkerStore,
     EndToEndRoomKeyStore,
     PresenceStore,
-    DeviceInboxWorkerStore,
+    SlavedDeviceInboxStore,
     SlavedDeviceStore,
+    SlavedReceiptsStore,
     SlavedPushRuleStore,
-    TagsWorkerStore,
-    AccountDataWorkerStore,
+    SlavedAccountDataStore,
     SlavedPusherStore,
     CensorEventsStore,
     ClientIpWorkerStore,
@@ -242,20 +239,19 @@ class GenericWorkerSlavedStore(
     SlavedKeyStore,
     RoomWorkerStore,
     RoomBatchStore,
-    DirectoryWorkerStore,
-    ApplicationServiceTransactionWorkerStore,
-    ApplicationServiceWorkerStore,
-    ProfileWorkerStore,
+    DirectoryStore,
+    SlavedApplicationServiceStore,
+    SlavedRegistrationStore,
+    SlavedProfileStore,
     SlavedFilteringStore,
     MonthlyActiveUsersWorkerStore,
     MediaRepositoryStore,
     ServerMetricsStore,
-    ReceiptsWorkerStore,
-    RegistrationWorkerStore,
     SearchStore,
     TransactionWorkerStore,
     LockStore,
     SessionStore,
+    BaseSlavedStore,
 ):
     # Properties that multiple storage classes define. Tell mypy what the
     # expected type is.

synapse/config/_base.py

@@ -97,16 +97,16 @@ def format_config_error(e: ConfigError) -> Iterator[str]:
 # We split these messages out to allow packages to override with package
 # specific instructions.
 MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS = """\
-Please opt in or out of reporting homeserver usage statistics, by setting
-the `report_stats` key in your config file to either True or False.
+Please opt in or out of reporting anonymized homeserver usage statistics, by
+setting the `report_stats` key in your config file to either True or False.
 """
 
 MISSING_REPORT_STATS_SPIEL = """\
 We would really appreciate it if you could help our project out by reporting
-homeserver usage statistics from your homeserver. Your homeserver's server name,
-along with very basic aggregate data (e.g. number of users) will be reported. But
-it helps us to track the growth of the Matrix community, and helps us to make Matrix
-a success, as well as to convince other networks that they should peer with us.
+anonymized usage statistics from your homeserver. Only very basic aggregate
+data (e.g. number of users) will be reported, but it helps us to track the
+growth of the Matrix community, and helps us to make Matrix a success, as well
+as to convince other networks that they should peer with us.
 
 Thank you.
 """
@@ -621,7 +621,7 @@ class RootConfig:
         generate_group.add_argument(
             "--report-stats",
             action="store",
-            help="Whether the generated config reports homeserver usage statistics.",
+            help="Whether the generated config reports anonymized usage statistics.",
             choices=["yes", "no"],
         )
         generate_group.add_argument(

synapse/config/ratelimiting.py

@@ -112,13 +112,6 @@ class RatelimitConfig(Config):
             defaults={"per_second": 0.01, "burst_count": 10},
         )
 
-        # Track the rate of joins to a given room. If there are too many, temporarily
-        # prevent local joins and remote joins via this server.
-        self.rc_joins_per_room = RateLimitConfig(
-            config.get("rc_joins_per_room", {}),
-            defaults={"per_second": 1, "burst_count": 10},
-        )
-
         # Ratelimit cross-user key requests:
         # * For local requests this is keyed by the sending device.
         # * For requests received over federation this is keyed by the origin.

synapse/config/repository.py

@@ -42,18 +42,6 @@ THUMBNAIL_SIZE_YAML = """\
         #    method: %(method)s
 """
 
-# A map from the given media type to the type of thumbnail we should generate
-# for it.
-THUMBNAIL_SUPPORTED_MEDIA_FORMAT_MAP = {
-    "image/jpeg": "jpeg",
-    "image/jpg": "jpeg",
-    "image/webp": "jpeg",
-    # Thumbnails can only be jpeg or png. We choose png thumbnails for gif
-    # because it can have transparency.
-    "image/gif": "png",
-    "image/png": "png",
-}
-
 HTTP_PROXY_SET_WARNING = """\
 The Synapse config url_preview_ip_range_blacklist will be ignored as an HTTP(s) proxy is configured."""
 
@@ -91,22 +79,13 @@ def parse_thumbnail_requirements(
         width = size["width"]
         height = size["height"]
         method = size["method"]
-
-        for format, thumbnail_format in THUMBNAIL_SUPPORTED_MEDIA_FORMAT_MAP.items():
-            requirement = requirements.setdefault(format, [])
-            if thumbnail_format == "jpeg":
-                requirement.append(
-                    ThumbnailRequirement(width, height, method, "image/jpeg")
-                )
-            elif thumbnail_format == "png":
-                requirement.append(
-                    ThumbnailRequirement(width, height, method, "image/png")
-                )
-            else:
-                raise Exception(
-                    "Unknown thumbnail mapping from %s to %s. This is a Synapse problem, please report!"
-                    % (format, thumbnail_format)
-                )
+        jpeg_thumbnail = ThumbnailRequirement(width, height, method, "image/jpeg")
+        png_thumbnail = ThumbnailRequirement(width, height, method, "image/png")
+        requirements.setdefault("image/jpeg", []).append(jpeg_thumbnail)
+        requirements.setdefault("image/jpg", []).append(jpeg_thumbnail)
+        requirements.setdefault("image/webp", []).append(jpeg_thumbnail)
+        requirements.setdefault("image/gif", []).append(png_thumbnail)
+        requirements.setdefault("image/png", []).append(png_thumbnail)
     return {
         media_type: tuple(thumbnails) for media_type, thumbnails in requirements.items()
     }

synapse/events/builder.py

@@ -24,11 +24,9 @@ from synapse.api.room_versions import (
     RoomVersion,
 )
 from synapse.crypto.event_signing import add_hashes_and_signatures
-from synapse.event_auth import auth_types_for_event
 from synapse.events import EventBase, _EventInternalMetadata, make_event_from_dict
 from synapse.state import StateHandler
 from synapse.storage.databases.main import DataStore
-from synapse.storage.state import StateFilter
 from synapse.types import EventID, JsonDict
 from synapse.util import Clock
 from synapse.util.stringutils import random_string
@@ -123,11 +121,7 @@ class EventBuilder:
         """
         if auth_event_ids is None:
             state_ids = await self._state.compute_state_after_events(
-                self.room_id,
-                prev_event_ids,
-                state_filter=StateFilter.from_types(
-                    auth_types_for_event(self.room_version, self)
-                ),
+                self.room_id, prev_event_ids
             )
             auth_event_ids = self._event_auth_handler.compute_auth_events(
                 self, state_ids

synapse/federation/federation_client.py

@@ -53,7 +53,7 @@ from synapse.api.room_versions import (
     RoomVersion,
     RoomVersions,
 )
-from synapse.events import EventBase, builder, make_event_from_dict
+from synapse.events import EventBase, builder
 from synapse.federation.federation_base import (
     FederationBase,
     InvalidEventSignatureError,
@@ -217,7 +217,7 @@ class FederationClient(FederationBase):
         )
 
     async def claim_client_keys(
-        self, destination: str, content: JsonDict, timeout: Optional[int]
+        self, destination: str, content: JsonDict, timeout: int
     ) -> JsonDict:
         """Claims one-time keys for a device hosted on a remote server.
 
@@ -299,8 +299,7 @@ class FederationClient(FederationBase):
                 moving to the next destination. None indicates no timeout.
 
         Returns:
-            A copy of the requested PDU that is safe to modify, or None if we
-            were unable to find it.
+            The requested PDU, or None if we were unable to find it.
 
         Raises:
             SynapseError, NotRetryingDestination, FederationDeniedError
@@ -310,7 +309,7 @@ class FederationClient(FederationBase):
         )
 
         logger.debug(
-            "get_pdu_from_destination_raw: retrieved event id %s from %s: %r",
+            "retrieved event id %s from %s: %r",
             event_id,
             destination,
             transaction_data,
@@ -359,92 +358,54 @@ class FederationClient(FederationBase):
             The requested PDU, or None if we were unable to find it.
         """
 
-        logger.debug(
-            "get_pdu: event_id=%s from destinations=%s", event_id, destinations
-        )
-
         # TODO: Rate limit the number of times we try and get the same event.
 
-        # We might need the same event multiple times in quick succession (before
-        # it gets persisted to the database), so we cache the results of the lookup.
-        # Note that this is separate to the regular get_event cache which caches
-        # events once they have been persisted.
-        event = self._get_pdu_cache.get(event_id)
+        ev = self._get_pdu_cache.get(event_id)
+        if ev:
+            return ev
 
-        # If we don't see the event in the cache, go try to fetch it from the
-        # provided remote federated destinations
-        if not event:
-            pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})
+        pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})
 
-            for destination in destinations:
-                now = self._clock.time_msec()
-                last_attempt = pdu_attempts.get(destination, 0)
-                if last_attempt + PDU_RETRY_TIME_MS > now:
-                    logger.debug(
-                        "get_pdu: skipping destination=%s because we tried it recently last_attempt=%s and we only check every %s (now=%s)",
-                        destination,
-                        last_attempt,
-                        PDU_RETRY_TIME_MS,
-                        now,
-                    )
-                    continue
+        signed_pdu = None
+        for destination in destinations:
+            now = self._clock.time_msec()
+            last_attempt = pdu_attempts.get(destination, 0)
+            if last_attempt + PDU_RETRY_TIME_MS > now:
+                continue
 
-                try:
-                    event = await self.get_pdu_from_destination_raw(
-                        destination=destination,
-                        event_id=event_id,
-                        room_version=room_version,
-                        timeout=timeout,
-                    )
+            try:
+                signed_pdu = await self.get_pdu_from_destination_raw(
+                    destination=destination,
+                    event_id=event_id,
+                    room_version=room_version,
+                    timeout=timeout,
+                )
 
-                    pdu_attempts[destination] = now
+                pdu_attempts[destination] = now
 
-                    if event:
-                        # Prime the cache
-                        self._get_pdu_cache[event.event_id] = event
+            except SynapseError as e:
+                logger.info(
+                    "Failed to get PDU %s from %s because %s", event_id, destination, e
+                )
+                continue
+            except NotRetryingDestination as e:
+                logger.info(str(e))
+                continue
+            except FederationDeniedError as e:
+                logger.info(str(e))
+                continue
+            except Exception as e:
+                pdu_attempts[destination] = now
 
-                        # FIXME: We should add a `break` here to avoid calling every
-                        # destination after we already found a PDU (will follow-up
-                        # in a separate PR)
+                logger.info(
+                    "Failed to get PDU %s from %s because %s", event_id, destination, e
+                )
+                continue
 
-                except SynapseError as e:
-                    logger.info(
-                        "Failed to get PDU %s from %s because %s",
-                        event_id,
-                        destination,
-                        e,
-                    )
-                    continue
-                except NotRetryingDestination as e:
-                    logger.info(str(e))
-                    continue
-                except FederationDeniedError as e:
-                    logger.info(str(e))
-                    continue
-                except Exception as e:
-                    pdu_attempts[destination] = now
+        if signed_pdu:
+            self._get_pdu_cache[event_id] = signed_pdu
 
-                    logger.info(
-                        "Failed to get PDU %s from %s because %s",
-                        event_id,
-                        destination,
-                        e,
-                    )
-                    continue
-
-        if not event:
-            return None
-
-        # `event` now refers to an object stored in `get_pdu_cache`. Our
-        # callers may need to modify the returned object (eg to set
-        # `event.internal_metadata.outlier = true`), so we return a copy
-        # rather than the original object.
-        event_copy = make_event_from_dict(
-            event.get_pdu_json(),
-            event.room_version,
-        )
-
-        return event_copy
+        return signed_pdu
 
     async def get_room_state_ids(
         self, destination: str, room_id: str, event_id: str

synapse/federation/federation_server.py

@@ -118,7 +118,6 @@ class FederationServer(FederationBase):
         self._federation_event_handler = hs.get_federation_event_handler()
         self.state = hs.get_state_handler()
         self._event_auth_handler = hs.get_event_auth_handler()
-        self._room_member_handler = hs.get_room_member_handler()
 
         self._state_storage_controller = hs.get_storage_controllers().state
 
@@ -622,15 +621,6 @@ class FederationServer(FederationBase):
             )
             raise IncompatibleRoomVersionError(room_version=room_version)
 
-        # Refuse the request if that room has seen too many joins recently.
-        # This is in addition to the HS-level rate limiting applied by
-        # BaseFederationServlet.
-        # type-ignore: mypy doesn't seem able to deduce the type of the limiter(!?)
-        await self._room_member_handler._join_rate_per_room_limiter.ratelimit(  # type: ignore[has-type]
-            requester=None,
-            key=room_id,
-            update=False,
-        )
         pdu = await self.handler.on_make_join_request(origin, room_id, user_id)
         return {"event": pdu.get_templated_pdu_json(), "room_version": room_version}
 
@@ -665,12 +655,6 @@ class FederationServer(FederationBase):
         room_id: str,
         caller_supports_partial_state: bool = False,
     ) -> Dict[str, Any]:
-        await self._room_member_handler._join_rate_per_room_limiter.ratelimit(  # type: ignore[has-type]
-            requester=None,
-            key=room_id,
-            update=False,
-        )
-
         event, context = await self._on_send_membership_event(
             origin, content, Membership.JOIN, room_id
         )

synapse/federation/transport/client.py

@@ -619,7 +619,7 @@ class TransportLayerClient:
         )
 
     async def claim_client_keys(
-        self, destination: str, query_content: JsonDict, timeout: Optional[int]
+        self, destination: str, query_content: JsonDict, timeout: int
     ) -> JsonDict:
         """Claim one-time keys for a list of devices hosted on a remote server.
 

synapse/federation/transport/server/_base.py

@@ -309,7 +309,7 @@ class BaseFederationServlet:
                 raise
 
             # update the active opentracing span with the authenticated entity
-            set_tag("authenticated_entity", str(origin))
+            set_tag("authenticated_entity", origin)
 
             # if the origin is authenticated and whitelisted, use its span context
             # as the parent.

synapse/handlers/device.py

@@ -118,8 +118,8 @@ class DeviceWorkerHandler:
         ips = await self.store.get_last_client_ip_by_device(user_id, device_id)
         _update_device_from_client_ips(device, ips)
 
-        set_tag("device", str(device))
-        set_tag("ips", str(ips))
+        set_tag("device", device)
+        set_tag("ips", ips)
 
         return device
 
@@ -170,7 +170,7 @@ class DeviceWorkerHandler:
         """
 
         set_tag("user_id", user_id)
-        set_tag("from_token", str(from_token))
+        set_tag("from_token", from_token)
         now_room_key = self.store.get_room_max_token()
 
         room_ids = await self.store.get_rooms_for_user(user_id)
@@ -795,7 +795,7 @@ class DeviceListUpdater:
         """
 
         set_tag("origin", origin)
-        set_tag("edu_content", str(edu_content))
+        set_tag("edu_content", edu_content)
         user_id = edu_content.pop("user_id")
         device_id = edu_content.pop("device_id")
         stream_id = str(edu_content.pop("stream_id"))  # They may come as ints

synapse/handlers/e2e_keys.py

@@ -15,7 +15,7 @@
 # limitations under the License.
 
 import logging
-from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Mapping, Optional, Tuple
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Tuple
 
 import attr
 from canonicaljson import encode_canonical_json
@@ -92,11 +92,7 @@ class E2eKeysHandler:
 
     @trace
     async def query_devices(
-        self,
-        query_body: JsonDict,
-        timeout: int,
-        from_user_id: str,
-        from_device_id: Optional[str],
+        self, query_body: JsonDict, timeout: int, from_user_id: str, from_device_id: str
     ) -> JsonDict:
         """Handle a device key query from a client
 
@@ -124,7 +120,9 @@ class E2eKeysHandler:
                 the number of in-flight queries at a time.
         """
         async with self._query_devices_linearizer.queue((from_user_id, from_device_id)):
-            device_keys_query: Dict[str, List[str]] = query_body.get("device_keys", {})
+            device_keys_query: Dict[str, Iterable[str]] = query_body.get(
+                "device_keys", {}
+            )
 
             # separate users by domain.
             # make a map from domain to user_id to device_ids
@@ -138,8 +136,8 @@ class E2eKeysHandler:
                 else:
                     remote_queries[user_id] = device_ids
 
-            set_tag("local_key_query", str(local_query))
-            set_tag("remote_key_query", str(remote_queries))
+            set_tag("local_key_query", local_query)
+            set_tag("remote_key_query", remote_queries)
 
             # First get local devices.
             # A map of destination -> failure response.
@@ -343,7 +341,7 @@ class E2eKeysHandler:
             failure = _exception_to_failure(e)
             failures[destination] = failure
             set_tag("error", True)
-            set_tag("reason", str(failure))
+            set_tag("reason", failure)
 
         return
 
@@ -394,7 +392,7 @@ class E2eKeysHandler:
 
     @trace
     async def query_local_devices(
-        self, query: Mapping[str, Optional[List[str]]]
+        self, query: Dict[str, Optional[List[str]]]
     ) -> Dict[str, Dict[str, dict]]:
         """Get E2E device keys for local users
 
@@ -405,7 +403,7 @@ class E2eKeysHandler:
         Returns:
             A map from user_id -> device_id -> device details
         """
-        set_tag("local_query", str(query))
+        set_tag("local_query", query)
         local_query: List[Tuple[str, Optional[str]]] = []
 
         result_dict: Dict[str, Dict[str, dict]] = {}
@@ -463,7 +461,7 @@ class E2eKeysHandler:
 
     @trace
     async def claim_one_time_keys(
-        self, query: Dict[str, Dict[str, Dict[str, str]]], timeout: Optional[int]
+        self, query: Dict[str, Dict[str, Dict[str, str]]], timeout: int
     ) -> JsonDict:
         local_query: List[Tuple[str, str, str]] = []
         remote_queries: Dict[str, Dict[str, Dict[str, str]]] = {}
@@ -477,8 +475,8 @@ class E2eKeysHandler:
                 domain = get_domain_from_id(user_id)
                 remote_queries.setdefault(domain, {})[user_id] = one_time_keys
 
-        set_tag("local_key_query", str(local_query))
-        set_tag("remote_key_query", str(remote_queries))
+        set_tag("local_key_query", local_query)
+        set_tag("remote_key_query", remote_queries)
 
         results = await self.store.claim_e2e_one_time_keys(local_query)
 
@@ -508,7 +506,7 @@ class E2eKeysHandler:
                 failure = _exception_to_failure(e)
                 failures[destination] = failure
                 set_tag("error", True)
-                set_tag("reason", str(failure))
+                set_tag("reason", failure)
 
         await make_deferred_yieldable(
             defer.gatherResults(
@@ -611,7 +609,7 @@ class E2eKeysHandler:
 
         result = await self.store.count_e2e_one_time_keys(user_id, device_id)
 
-        set_tag("one_time_key_counts", str(result))
+        set_tag("one_time_key_counts", result)
         return {"one_time_key_counts": result}
 
     async def _upload_one_time_keys_for_user(

synapse/handlers/e2e_room_keys.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 import logging
-from typing import TYPE_CHECKING, Dict, Optional, cast
+from typing import TYPE_CHECKING, Dict, Optional
 
 from typing_extensions import Literal
 
@@ -97,7 +97,7 @@ class E2eRoomKeysHandler:
                 user_id, version, room_id, session_id
             )
 
-            log_kv(cast(JsonDict, results))
+            log_kv(results)
             return results
 
     @trace

synapse/handlers/federation_event.py

@@ -766,24 +766,10 @@ class FederationEventHandler:
         """
         logger.info("Processing pulled event %s", event)
 
-        # This function should not be used to persist outliers (use something
-        # else) because this does a bunch of operations that aren't necessary
-        # (extra work; in particular, it makes sure we have all the prev_events
-        # and resolves the state across those prev events). If you happen to run
-        # into a situation where the event you're trying to process/backfill is
-        # marked as an `outlier`, then you should update that spot to return an
-        # `EventBase` copy that doesn't have `outlier` flag set.
-        #
-        # `EventBase` is used to represent both an event we have not yet
-        # persisted, and one that we have persisted and now keep in the cache.
-        # In an ideal world this method would only be called with the first type
-        # of event, but it turns out that's not actually the case and for
-        # example, you could get an event from cache that is marked as an
-        # `outlier` (fix up that spot though).
-        assert not event.internal_metadata.is_outlier(), (
-            "Outlier event passed to _process_pulled_event. "
-            "To persist an event as a non-outlier, make sure to pass in a copy without `event.internal_metadata.outlier = true`."
-        )
+        # these should not be outliers.
+        assert (
+            not event.internal_metadata.is_outlier()
+        ), "pulled event unexpectedly flagged as outlier"
 
         event_id = event.event_id
 
@@ -793,7 +779,7 @@ class FederationEventHandler:
         if existing:
             if not existing.internal_metadata.is_outlier():
                 logger.info(
-                    "_process_pulled_event: Ignoring received event %s which we have already seen",
+                    "Ignoring received event %s which we have already seen",
                     event_id,
                 )
                 return
@@ -1051,9 +1037,6 @@ class FederationEventHandler:
         # XXX: this doesn't sound right? it means that we'll end up with incomplete
         #   state.
         failed_to_fetch = desired_events - event_metadata.keys()
-        # `event_id` could be missing from `event_metadata` because it's not necessarily
-        # a state event. We've already checked that we've fetched it above.
-        failed_to_fetch.discard(event_id)
         if failed_to_fetch:
             logger.warning(
                 "Failed to fetch missing state events for %s %s",
@@ -1329,53 +1312,6 @@ class FederationEventHandler:
             marker_event,
         )
 
-    async def backfill_event_id(
-        self, destination: str, room_id: str, event_id: str
-    ) -> EventBase:
-        """Backfill a single event and persist it as a non-outlier which means
-        we also pull in all of the state and auth events necessary for it.
-
-        Args:
-            destination: The homeserver to pull the given event_id from.
-            room_id: The room where the event is from.
-            event_id: The event ID to backfill.
-
-        Raises:
-            FederationError if we are unable to find the event from the destination
-        """
-        logger.info(
-            "backfill_event_id: event_id=%s from destination=%s", event_id, destination
-        )
-
-        room_version = await self._store.get_room_version(room_id)
-
-        event_from_response = await self._federation_client.get_pdu(
-            [destination],
-            event_id,
-            room_version,
-        )
-
-        if not event_from_response:
-            raise FederationError(
-                "ERROR",
-                404,
-                "Unable to find event_id=%s from destination=%s to backfill."
-                % (event_id, destination),
-                affected=event_id,
-            )
-
-        # Persist the event we just fetched, including pulling all of the state
-        # and auth events to de-outlier it. This also sets up the necessary
-        # `state_groups` for the event.
-        await self._process_pulled_events(
-            destination,
-            [event_from_response],
-            # Prevent notifications going to clients
-            backfilled=True,
-        )
-
-        return event_from_response
-
     async def _get_events_and_persist(
         self, destination: str, room_id: str, event_ids: Collection[str]
     ) -> None:
@@ -1711,21 +1647,11 @@ class FederationEventHandler:
         """Checks if we should soft fail the event; if so, marks the event as
         such.
 
-        Does nothing for events in rooms with partial state, since we may not have an
-        accurate membership event for the sender in the current state.
-
         Args:
             event
             state_ids: The state at the event if we don't have all the event's prev events
             origin: The host the event originates from.
         """
-        if await self._store.is_partial_state_room(event.room_id):
-            # We might not know the sender's membership in the current state, so don't
-            # soft fail anything. Even if we do have a membership for the sender in the
-            # current state, it may have been derived from state resolution between
-            # partial and full state and may not be accurate.
-            return
-
         extrem_ids_list = await self._store.get_latest_event_ids_in_room(event.room_id)
         extrem_ids = set(extrem_ids_list)
         prev_event_ids = set(event.prev_event_ids())
@@ -2054,10 +1980,6 @@ class FederationEventHandler:
             event, event_pos, max_stream_token, extra_users=extra_users
         )
 
-        if event.type == EventTypes.Member and event.membership == Membership.JOIN:
-            # TODO retrieve the previous state, and exclude join -> join transitions
-            self._notifier.notify_user_joined_room(event.event_id, event.room_id)
-
     def _sanity_check_event(self, ev: EventBase) -> None:
         """
         Do some early sanity checks of a received event

synapse/handlers/message.py

@@ -463,7 +463,6 @@ class EventCreationHandler:
         )
         self._events_shard_config = self.config.worker.events_shard_config
         self._instance_name = hs.get_instance_name()
-        self._notifier = hs.get_notifier()
 
         self.room_prejoin_state_types = self.hs.config.api.room_prejoin_state
 
@@ -1551,16 +1550,6 @@ class EventCreationHandler:
                 requester, is_admin_redaction=is_admin_redaction
             )
 
-        if event.type == EventTypes.Member and event.membership == Membership.JOIN:
-            (
-                current_membership,
-                _,
-            ) = await self.store.get_local_current_membership_for_user_in_room(
-                event.state_key, event.room_id
-            )
-            if current_membership != Membership.JOIN:
-                self._notifier.notify_user_joined_room(event.event_id, event.room_id)
-
         await self._maybe_kick_guest_users(event, context)
 
         if event.type == EventTypes.CanonicalAlias:
@@ -1860,8 +1849,13 @@ class EventCreationHandler:
 
         # For each room we need to find a joined member we can use to send
         # the dummy event with.
-        members = await self.store.get_local_users_in_room(room_id)
+        latest_event_ids = await self.store.get_prev_events_for_room(room_id)
+        members = await self.state.get_current_users_in_room(
+            room_id, latest_event_ids=latest_event_ids
+        )
         for user_id in members:
+            if not self.hs.is_mine_id(user_id):
+                continue
             requester = create_requester(user_id, authenticated_entity=self.server_name)
             try:
                 event, context = await self.create_event(
@@ -1872,6 +1866,7 @@ class EventCreationHandler:
                         "room_id": room_id,
                         "sender": user_id,
                     },
+                    prev_event_ids=latest_event_ids,
                 )
 
                 event.internal_metadata.proactively_send = False

synapse/handlers/presence.py

@@ -34,6 +34,7 @@ from typing import (
     Callable,
     Collection,
     Dict,
+    FrozenSet,
     Generator,
     Iterable,
     List,
@@ -41,6 +42,7 @@ from typing import (
     Set,
     Tuple,
     Type,
+    Union,
 )
 
 from prometheus_client import Counter
@@ -66,6 +68,7 @@ from synapse.storage.databases.main import DataStore
 from synapse.streams import EventSource
 from synapse.types import JsonDict, StreamKeyType, UserID, get_domain_from_id
 from synapse.util.async_helpers import Linearizer
+from synapse.util.caches.descriptors import _CacheContext, cached
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
 
@@ -1653,18 +1656,15 @@ class PresenceEventSource(EventSource[int, UserPresenceState]):
                 # doesn't return. C.f. #5503.
                 return [], max_token
 
-            # Figure out which other users this user should explicitly receive
-            # updates for
-            additional_users_interested_in = (
-                await self.get_presence_router().get_interested_users(user.to_string())
-            )
+            # Figure out which other users this user should receive updates for
+            users_interested_in = await self._get_interested_in(user, explicit_room_id)
 
             # We have a set of users that we're interested in the presence of. We want to
             # cross-reference that with the users that have actually changed their presence.
 
             # Check whether this user should see all user updates
 
-            if additional_users_interested_in == PresenceRouter.ALL_USERS:
+            if users_interested_in == PresenceRouter.ALL_USERS:
                 # Provide presence state for all users
                 presence_updates = await self._filter_all_presence_updates_for_user(
                     user_id, include_offline, from_key
@@ -1673,47 +1673,34 @@ class PresenceEventSource(EventSource[int, UserPresenceState]):
                 return presence_updates, max_token
 
             # Make mypy happy. users_interested_in should now be a set
-            assert not isinstance(additional_users_interested_in, str)
-
-            # We always care about our own presence.
-            additional_users_interested_in.add(user_id)
-
-            if explicit_room_id:
-                user_ids = await self.store.get_users_in_room(explicit_room_id)
-                additional_users_interested_in.update(user_ids)
+            assert not isinstance(users_interested_in, str)
 
             # The set of users that we're interested in and that have had a presence update.
             # We'll actually pull the presence updates for these users at the end.
-            interested_and_updated_users: Collection[str]
+            interested_and_updated_users: Union[Set[str], FrozenSet[str]] = set()
 
             if from_key is not None:
                 # First get all users that have had a presence update
                 updated_users = stream_change_cache.get_all_entities_changed(from_key)
 
                 # Cross-reference users we're interested in with those that have had updates.
-                if updated_users is not None:
-                    # If we have the full list of changes for presence we can
-                    # simply check which ones share a room with the user.
+                # Use a slightly-optimised method for processing smaller sets of updates.
+                if updated_users is not None and len(updated_users) < 500:
+                    # For small deltas, it's quicker to get all changes and then
+                    # cross-reference with the users we're interested in
                     get_updates_counter.labels("stream").inc()
-
-                    sharing_users = await self.store.do_users_share_a_room(
-                        user_id, updated_users
-                    )
-
-                    interested_and_updated_users = (
-                        sharing_users.union(additional_users_interested_in)
-                    ).intersection(updated_users)
-
+                    for other_user_id in updated_users:
+                        if other_user_id in users_interested_in:
+                            # mypy thinks this variable could be a FrozenSet as it's possibly set
+                            # to one in the `get_entities_changed` call below, and `add()` is not
+                            # method on a FrozenSet. That doesn't affect us here though, as
+                            # `interested_and_updated_users` is clearly a set() above.
+                            interested_and_updated_users.add(other_user_id)  # type: ignore
                 else:
                     # Too many possible updates. Find all users we can see and check
                     # if any of them have changed.
                     get_updates_counter.labels("full").inc()
 
-                    users_interested_in = (
-                        await self.store.get_users_who_share_room_with_user(user_id)
-                    )
-                    users_interested_in.update(additional_users_interested_in)
-
                     interested_and_updated_users = (
                         stream_change_cache.get_entities_changed(
                             users_interested_in, from_key
@@ -1722,10 +1709,7 @@ class PresenceEventSource(EventSource[int, UserPresenceState]):
             else:
                 # No from_key has been specified. Return the presence for all users
                 # this user is interested in
-                interested_and_updated_users = (
-                    await self.store.get_users_who_share_room_with_user(user_id)
-                )
-                interested_and_updated_users.update(additional_users_interested_in)
+                interested_and_updated_users = users_interested_in
 
             # Retrieve the current presence state for each user
             users_to_state = await self.get_presence_handler().current_state_for_users(
@@ -1820,6 +1804,62 @@ class PresenceEventSource(EventSource[int, UserPresenceState]):
     def get_current_key(self) -> int:
         return self.store.get_current_presence_token()
 
+    @cached(num_args=2, cache_context=True)
+    async def _get_interested_in(
+        self,
+        user: UserID,
+        explicit_room_id: Optional[str] = None,
+        cache_context: Optional[_CacheContext] = None,
+    ) -> Union[Set[str], str]:
+        """Returns the set of users that the given user should see presence
+        updates for.
+
+        Args:
+            user: The user to retrieve presence updates for.
+            explicit_room_id: The users that are in the room will be returned.
+
+        Returns:
+            A set of user IDs to return presence updates for, or "ALL" to return all
+            known updates.
+        """
+        user_id = user.to_string()
+        users_interested_in = set()
+        users_interested_in.add(user_id)  # So that we receive our own presence
+
+        # cache_context isn't likely to ever be None due to the @cached decorator,
+        # but we can't have a non-optional argument after the optional argument
+        # explicit_room_id either. Assert cache_context is not None so we can use it
+        # without mypy complaining.
+        assert cache_context
+
+        # Check with the presence router whether we should poll additional users for
+        # their presence information
+        additional_users = await self.get_presence_router().get_interested_users(
+            user.to_string()
+        )
+        if additional_users == PresenceRouter.ALL_USERS:
+            # If the module requested that this user see the presence updates of *all*
+            # users, then simply return that instead of calculating what rooms this
+            # user shares
+            return PresenceRouter.ALL_USERS
+
+        # Add the additional users from the router
+        users_interested_in.update(additional_users)
+
+        # Find the users who share a room with this user
+        users_who_share_room = await self.store.get_users_who_share_room_with_user(
+            user_id, on_invalidate=cache_context.invalidate
+        )
+        users_interested_in.update(users_who_share_room)
+
+        if explicit_room_id:
+            user_ids = await self.store.get_users_in_room(
+                explicit_room_id, on_invalidate=cache_context.invalidate
+            )
+            users_interested_in.update(user_ids)
+
+        return users_interested_in
+
 
 def handle_timeouts(
     user_states: List[UserPresenceState],

synapse/handlers/room.py

@@ -1384,7 +1384,6 @@ class TimestampLookupHandler:
         self.store = hs.get_datastores().main
         self.state_handler = hs.get_state_handler()
         self.federation_client = hs.get_federation_client()
-        self.federation_event_handler = hs.get_federation_event_handler()
         self._storage_controllers = hs.get_storage_controllers()
 
     async def get_event_for_timestamp(
@@ -1480,68 +1479,38 @@ class TimestampLookupHandler:
                         remote_response,
                     )
 
+                    # TODO: Do we want to persist this as an extremity?
+                    # TODO: I think ideally, we would try to backfill from
+                    # this event and run this whole
+                    # `get_event_for_timestamp` function again to make sure
+                    # they didn't give us an event from their gappy history.
                     remote_event_id = remote_response.event_id
-                    remote_origin_server_ts = remote_response.origin_server_ts
-
-                    # Backfill this event so we can get a pagination token for
-                    # it with `/context` and paginate `/messages` from this
-                    # point.
-                    #
-                    # TODO: The requested timestamp may lie in a part of the
-                    #   event graph that the remote server *also* didn't have,
-                    #   in which case they will have returned another event
-                    #   which may be nowhere near the requested timestamp. In
-                    #   the future, we may need to reconcile that gap and ask
-                    #   other homeservers, and/or extend `/timestamp_to_event`
-                    #   to return events on *both* sides of the timestamp to
-                    #   help reconcile the gap faster.
-                    remote_event = (
-                        await self.federation_event_handler.backfill_event_id(
-                            domain, room_id, remote_event_id
-                        )
-                    )
-
-                    # XXX: When we see that the remote server is not trustworthy,
-                    # maybe we should not ask them first in the future.
-                    if remote_origin_server_ts != remote_event.origin_server_ts:
-                        logger.info(
-                            "get_event_for_timestamp: Remote server (%s) claimed that remote_event_id=%s occured at remote_origin_server_ts=%s but that isn't true (actually occured at %s). Their claims are dubious and we should consider not trusting them.",
-                            domain,
-                            remote_event_id,
-                            remote_origin_server_ts,
-                            remote_event.origin_server_ts,
-                        )
+                    origin_server_ts = remote_response.origin_server_ts
 
                     # Only return the remote event if it's closer than the local event
                     if not local_event or (
-                        abs(remote_event.origin_server_ts - timestamp)
+                        abs(origin_server_ts - timestamp)
                         < abs(local_event.origin_server_ts - timestamp)
                     ):
-                        logger.info(
-                            "get_event_for_timestamp: returning remote_event_id=%s (%s) since it's closer to timestamp=%s than local_event=%s (%s)",
-                            remote_event_id,
-                            remote_event.origin_server_ts,
-                            timestamp,
-                            local_event.event_id if local_event else None,
-                            local_event.origin_server_ts if local_event else None,
-                        )
-                        return remote_event_id, remote_origin_server_ts
+                        return remote_event_id, origin_server_ts
                 except (HttpResponseException, InvalidResponseError) as ex:
                     # Let's not put a high priority on some other homeserver
                     # failing to respond or giving a random response
                     logger.debug(
-                        "get_event_for_timestamp: Failed to fetch /timestamp_to_event from %s because of exception(%s) %s args=%s",
+                        "Failed to fetch /timestamp_to_event from %s because of exception(%s) %s args=%s",
                         domain,
                         type(ex).__name__,
                         ex,
                         ex.args,
                     )
-                except Exception:
+                except Exception as ex:
                     # But we do want to see some exceptions in our code
                     logger.warning(
-                        "get_event_for_timestamp: Failed to fetch /timestamp_to_event from %s because of exception",
+                        "Failed to fetch /timestamp_to_event from %s because of exception(%s) %s args=%s",
                         domain,
-                        exc_info=True,
+                        type(ex).__name__,
+                        ex,
+                        ex.args,
                     )
 
         # To appease mypy, we have to add both of these conditions to check for

synapse/handlers/room_member.py

@@ -94,29 +94,12 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
             rate_hz=hs.config.ratelimiting.rc_joins_local.per_second,
             burst_count=hs.config.ratelimiting.rc_joins_local.burst_count,
         )
-        # Tracks joins from local users to rooms this server isn't a member of.
-        # I.e. joins this server makes by requesting /make_join /send_join from
-        # another server.
         self._join_rate_limiter_remote = Ratelimiter(
             store=self.store,
             clock=self.clock,
             rate_hz=hs.config.ratelimiting.rc_joins_remote.per_second,
             burst_count=hs.config.ratelimiting.rc_joins_remote.burst_count,
         )
-        # TODO: find a better place to keep this Ratelimiter.
-        #   It needs to be
-        #    - written to by event persistence code
-        #    - written to by something which can snoop on replication streams
-        #    - read by the RoomMemberHandler to rate limit joins from local users
-        #    - read by the FederationServer to rate limit make_joins and send_joins from
-        #      other homeservers
-        #   I wonder if a homeserver-wide collection of rate limiters might be cleaner?
-        self._join_rate_per_room_limiter = Ratelimiter(
-            store=self.store,
-            clock=self.clock,
-            rate_hz=hs.config.ratelimiting.rc_joins_per_room.per_second,
-            burst_count=hs.config.ratelimiting.rc_joins_per_room.burst_count,
-        )
 
         # Ratelimiter for invites, keyed by room (across all issuers, all
         # recipients).
@@ -153,18 +136,6 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
         )
 
         self.request_ratelimiter = hs.get_request_ratelimiter()
-        hs.get_notifier().add_new_join_in_room_callback(self._on_user_joined_room)
-
-    def _on_user_joined_room(self, event_id: str, room_id: str) -> None:
-        """Notify the rate limiter that a room join has occurred.
-
-        Use this to inform the RoomMemberHandler about joins that have either
-        - taken place on another homeserver, or
-        - on another worker in this homeserver.
-        Joins actioned by this worker should use the usual `ratelimit` method, which
-        checks the limit and increments the counter in one go.
-        """
-        self._join_rate_per_room_limiter.record_action(requester=None, key=room_id)
 
     @abc.abstractmethod
     async def _remote_join(
@@ -425,9 +396,6 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
             # up blocking profile updates.
             if newly_joined and ratelimit:
                 await self._join_rate_limiter_local.ratelimit(requester)
-                await self._join_rate_per_room_limiter.ratelimit(
-                    requester, key=room_id, update=False
-                )
 
         result_event = await self.event_creation_handler.handle_new_client_event(
             requester,
@@ -899,11 +867,6 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                     await self._join_rate_limiter_remote.ratelimit(
                         requester,
                     )
-                    await self._join_rate_per_room_limiter.ratelimit(
-                        requester,
-                        key=room_id,
-                        update=False,
-                    )
 
                 inviter = await self._get_inviter(target.to_string(), room_id)
                 if inviter and not self.hs.is_mine(inviter):

synapse/http/matrixfederationclient.py

@@ -79,7 +79,6 @@ from synapse.types import JsonDict
 from synapse.util import json_decoder
 from synapse.util.async_helpers import AwakenableSleeper, timeout_deferred
 from synapse.util.metrics import Measure
-from synapse.util.stringutils import parse_and_validate_server_name
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -480,14 +479,6 @@ class MatrixFederationHttpClient:
             RequestSendFailed: If there were problems connecting to the
                 remote, due to e.g. DNS failures, connection timeouts etc.
         """
-        # Validate server name and log if it is an invalid destination, this is
-        # partially to help track down code paths where we haven't validated before here
-        try:
-            parse_and_validate_server_name(request.destination)
-        except ValueError:
-            logger.exception(f"Invalid destination: {request.destination}.")
-            raise FederationDeniedError(request.destination)
-
         if timeout:
             _sec_timeout = timeout / 1000
         else:

synapse/logging/opentracing.py

@@ -84,13 +84,14 @@ the function becomes the operation name for the span.
        return something_usual_and_useful
 
 
-Operation names can be explicitly set for a function by using ``trace_with_opname``:
+Operation names can be explicitly set for a function by passing the
+operation name to ``trace``
 
 .. code-block:: python
 
-   from synapse.logging.opentracing import trace_with_opname
+   from synapse.logging.opentracing import trace
 
-   @trace_with_opname("a_better_operation_name")
+   @trace(opname="a_better_operation_name")
    def interesting_badly_named_function(*args, **kwargs):
        # Does all kinds of cool and expected things
        return something_usual_and_useful
@@ -182,8 +183,6 @@ from typing import (
     Type,
     TypeVar,
     Union,
-    cast,
-    overload,
 )
 
 import attr
@@ -330,7 +329,6 @@ class _Sentinel(enum.Enum):
 
 P = ParamSpec("P")
 R = TypeVar("R")
-T = TypeVar("T")
 
 
 def only_if_tracing(func: Callable[P, R]) -> Callable[P, Optional[R]]:
@@ -346,43 +344,22 @@ def only_if_tracing(func: Callable[P, R]) -> Callable[P, Optional[R]]:
     return _only_if_tracing_inner
 
 
-@overload
-def ensure_active_span(
-    message: str,
-) -> Callable[[Callable[P, R]], Callable[P, Optional[R]]]:
-    ...
-
-
-@overload
-def ensure_active_span(
-    message: str, ret: T
-) -> Callable[[Callable[P, R]], Callable[P, Union[T, R]]]:
-    ...
-
-
-def ensure_active_span(
-    message: str, ret: Optional[T] = None
-) -> Callable[[Callable[P, R]], Callable[P, Union[Optional[T], R]]]:
+def ensure_active_span(message: str, ret=None):
     """Executes the operation only if opentracing is enabled and there is an active span.
     If there is no active span it logs message at the error level.
 
     Args:
         message: Message which fills in "There was no active span when trying to %s"
             in the error log if there is no active span and opentracing is enabled.
-        ret: return value if opentracing is None or there is no active span.
+        ret (object): return value if opentracing is None or there is no active span.
 
-    Returns:
-        The result of the func, falling back to ret if opentracing is disabled or there
+    Returns (object): The result of the func or ret if opentracing is disabled or there
         was no active span.
     """
 
-    def ensure_active_span_inner_1(
-        func: Callable[P, R]
-    ) -> Callable[P, Union[Optional[T], R]]:
+    def ensure_active_span_inner_1(func):
         @wraps(func)
-        def ensure_active_span_inner_2(
-            *args: P.args, **kwargs: P.kwargs
-        ) -> Union[Optional[T], R]:
+        def ensure_active_span_inner_2(*args, **kwargs):
             if not opentracing:
                 return ret
 
@@ -488,7 +465,7 @@ def start_active_span(
     finish_on_close: bool = True,
     *,
     tracer: Optional["opentracing.Tracer"] = None,
-) -> "opentracing.Scope":
+):
     """Starts an active opentracing span.
 
     Records the start time for the span, and sets it as the "active span" in the
@@ -526,7 +503,7 @@ def start_active_span_follows_from(
     *,
     inherit_force_tracing: bool = False,
     tracer: Optional["opentracing.Tracer"] = None,
-) -> "opentracing.Scope":
+):
     """Starts an active opentracing span, with additional references to previous spans
 
     Args:
@@ -741,9 +718,7 @@ def inject_response_headers(response_headers: Headers) -> None:
         response_headers.addRawHeader("Synapse-Trace-Id", f"{trace_id:x}")
 
 
-@ensure_active_span(
-    "get the active span context as a dict", ret=cast(Dict[str, str], {})
-)
+@ensure_active_span("get the active span context as a dict", ret={})
 def get_active_span_text_map(destination: Optional[str] = None) -> Dict[str, str]:
     """
     Gets a span context as a dict. This can be used instead of manually
@@ -823,31 +798,33 @@ def extract_text_map(carrier: Dict[str, str]) -> Optional["opentracing.SpanConte
 # Tracing decorators
 
 
-def trace_with_opname(opname: str) -> Callable[[Callable[P, R]], Callable[P, R]]:
+def trace(func=None, opname: Optional[str] = None):
     """
-    Decorator to trace a function with a custom opname.
-
-    See the module's doc string for usage examples.
-
+    Decorator to trace a function.
+    Sets the operation name to that of the function's or that given
+    as operation_name. See the module's doc string for usage
+    examples.
     """
 
-    def decorator(func: Callable[P, R]) -> Callable[P, R]:
+    def decorator(func):
         if opentracing is None:
             return func  # type: ignore[unreachable]
 
+        _opname = opname if opname else func.__name__
+
         if inspect.iscoroutinefunction(func):
 
             @wraps(func)
-            async def _trace_inner(*args: P.args, **kwargs: P.kwargs) -> R:
-                with start_active_span(opname):
-                    return await func(*args, **kwargs)  # type: ignore[misc]
+            async def _trace_inner(*args, **kwargs):
+                with start_active_span(_opname):
+                    return await func(*args, **kwargs)
 
         else:
             # The other case here handles both sync functions and those
             # decorated with inlineDeferred.
             @wraps(func)
-            def _trace_inner(*args: P.args, **kwargs: P.kwargs) -> R:
-                scope = start_active_span(opname)
+            def _trace_inner(*args, **kwargs):
+                scope = start_active_span(_opname)
                 scope.__enter__()
 
                 try:
@@ -881,21 +858,12 @@ def trace_with_opname(opname: str) -> Callable[[Callable[P, R]], Callable[P, R]]
                     scope.__exit__(type(e), None, e.__traceback__)
                     raise
 
-        return _trace_inner  # type: ignore[return-value]
+        return _trace_inner
 
-    return decorator
-
-
-def trace(func: Callable[P, R]) -> Callable[P, R]:
-    """
-    Decorator to trace a function.
-
-    Sets the operation name to that of the function's name.
-
-    See the module's doc string for usage examples.
-    """
-
-    return trace_with_opname(func.__name__)(func)
+    if func:
+        return decorator(func)
+    else:
+        return decorator
 
 
 def tag_args(func: Callable[P, R]) -> Callable[P, R]:
@@ -910,9 +878,9 @@ def tag_args(func: Callable[P, R]) -> Callable[P, R]:
     def _tag_args_inner(*args: P.args, **kwargs: P.kwargs) -> R:
         argspec = inspect.getfullargspec(func)
         for i, arg in enumerate(argspec.args[1:]):
-            set_tag("ARG_" + arg, str(args[i]))  # type: ignore[index]
-        set_tag("args", str(args[len(argspec.args) :]))  # type: ignore[index]
-        set_tag("kwargs", str(kwargs))
+            set_tag("ARG_" + arg, args[i])  # type: ignore[index]
+        set_tag("args", args[len(argspec.args) :])  # type: ignore[index]
+        set_tag("kwargs", kwargs)
         return func(*args, **kwargs)
 
     return _tag_args_inner

synapse/metrics/background_process_metrics.py

@@ -235,7 +235,7 @@ def run_as_background_process(
                         f"bgproc.{desc}", tags={SynapseTags.REQUEST_ID: str(context)}
                     )
                 else:
-                    ctx = nullcontext()  # type: ignore[assignment]
+                    ctx = nullcontext()
                 with ctx:
                     return await func(*args, **kwargs)
             except Exception:

synapse/push/bulk_push_rule_evaluator.py

@@ -131,13 +131,6 @@ class BulkPushRuleEvaluator:
 
         local_users = await self.store.get_local_users_in_room(event.room_id)
 
-        # Filter out appservice users.
-        local_users = [
-            u
-            for u in local_users
-            if not self.store.get_if_app_services_interested_in_user(u)
-        ]
-
         # if this event is an invite event, we may need to run rules for the user
         # who's been invited, otherwise they won't get told they've been invited
         if event.type == EventTypes.Member and event.membership == Membership.INVITE:

synapse/push/pusherpool.py

@@ -328,7 +328,7 @@ class PusherPool:
             return None
 
         try:
-            pusher = self.pusher_factory.create_pusher(pusher_config)
+            p = self.pusher_factory.create_pusher(pusher_config)
         except PusherConfigException as e:
             logger.warning(
                 "Pusher incorrectly configured id=%i, user=%s, appid=%s, pushkey=%s: %s",
@@ -346,28 +346,23 @@ class PusherPool:
             )
             return None
 
-        if not pusher:
+        if not p:
             return None
 
-        appid_pushkey = "%s:%s" % (pusher.app_id, pusher.pushkey)
+        appid_pushkey = "%s:%s" % (pusher_config.app_id, pusher_config.pushkey)
 
-        byuser = self.pushers.setdefault(pusher.user_id, {})
+        byuser = self.pushers.setdefault(pusher_config.user_name, {})
         if appid_pushkey in byuser:
-            previous_pusher = byuser[appid_pushkey]
-            previous_pusher.on_stop()
+            byuser[appid_pushkey].on_stop()
+        byuser[appid_pushkey] = p
 
-            synapse_pushers.labels(
-                type(previous_pusher).__name__, previous_pusher.app_id
-            ).dec()
-        byuser[appid_pushkey] = pusher
-
-        synapse_pushers.labels(type(pusher).__name__, pusher.app_id).inc()
+        synapse_pushers.labels(type(p).__name__, p.app_id).inc()
 
         # Check if there *may* be push to process. We do this as this check is a
         # lot cheaper to do than actually fetching the exact rows we need to
         # push.
-        user_id = pusher.user_id
-        last_stream_ordering = pusher.last_stream_ordering
+        user_id = pusher_config.user_name
+        last_stream_ordering = pusher_config.last_stream_ordering
         if last_stream_ordering:
             have_notifs = await self.store.get_if_maybe_push_in_range_for_user(
                 user_id, last_stream_ordering
@@ -377,9 +372,9 @@ class PusherPool:
             # risk missing push.
             have_notifs = True
 
-        pusher.on_started(have_notifs)
+        p.on_started(have_notifs)
 
-        return pusher
+        return p
 
     async def remove_pusher(self, app_id: str, pushkey: str, user_id: str) -> None:
         appid_pushkey = "%s:%s" % (app_id, pushkey)

synapse/replication/http/_base.py

@@ -29,7 +29,7 @@ from synapse.http import RequestTimedOutError
 from synapse.http.server import HttpServer, is_method_cancellable
 from synapse.http.site import SynapseRequest
 from synapse.logging import opentracing
-from synapse.logging.opentracing import trace_with_opname
+from synapse.logging.opentracing import trace
 from synapse.types import JsonDict
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.util.stringutils import random_string
@@ -196,7 +196,7 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
                 "ascii"
             )
 
-        @trace_with_opname("outgoing_replication_request")
+        @trace(opname="outgoing_replication_request")
         async def send_request(*, instance_name: str = "master", **kwargs: Any) -> Any:
             with outgoing_gauge.track_inprogress():
                 if instance_name == local_instance_name:

synapse/replication/slave/storage/_base.py

@@ -0,0 +1,58 @@
+# Copyright 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import TYPE_CHECKING, Optional
+
+from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
+from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
+from synapse.storage.engines import PostgresEngine
+from synapse.storage.util.id_generators import MultiWriterIdGenerator
+
+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
+logger = logging.getLogger(__name__)
+
+
+class BaseSlavedStore(CacheInvalidationWorkerStore):
+    def __init__(
+        self,
+        database: DatabasePool,
+        db_conn: LoggingDatabaseConnection,
+        hs: "HomeServer",
+    ):
+        super().__init__(database, db_conn, hs)
+        if isinstance(self.database_engine, PostgresEngine):
+            self._cache_id_gen: Optional[
+                MultiWriterIdGenerator
+            ] = MultiWriterIdGenerator(
+                db_conn,
+                database,
+                stream_name="caches",
+                instance_name=hs.get_instance_name(),
+                tables=[
+                    (
+                        "cache_invalidation_stream_by_instance",
+                        "instance_name",
+                        "stream_id",
+                    )
+                ],
+                sequence_name="cache_invalidation_stream_seq",
+                writers=[],
+            )
+        else:
+            self._cache_id_gen = None
+
+        self.hs = hs

synapse/replication/slave/storage/account_data.py

@@ -0,0 +1,22 @@
+# Copyright 2016 OpenMarket Ltd
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.storage.databases.main.account_data import AccountDataWorkerStore
+from synapse.storage.databases.main.tags import TagsWorkerStore
+
+
+class SlavedAccountDataStore(TagsWorkerStore, AccountDataWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/slave/storage/appservice.py

@@ -0,0 +1,25 @@
+# Copyright 2015, 2016 OpenMarket Ltd
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.storage.databases.main.appservice import (
+    ApplicationServiceTransactionWorkerStore,
+    ApplicationServiceWorkerStore,
+)
+
+
+class SlavedApplicationServiceStore(
+    ApplicationServiceTransactionWorkerStore, ApplicationServiceWorkerStore
+):
+    pass

synapse/replication/slave/storage/deviceinbox.py

@@ -0,0 +1,20 @@
+# Copyright 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
+
+
+class SlavedDeviceInboxStore(DeviceInboxWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/slave/storage/devices.py

@@ -14,6 +14,7 @@
 
 from typing import TYPE_CHECKING, Any, Iterable
 
+from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
 from synapse.replication.tcp.streams._base import DeviceListsStream, UserSignatureStream
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
@@ -23,7 +24,7 @@ if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 
-class SlavedDeviceStore(DeviceWorkerStore):
+class SlavedDeviceStore(DeviceWorkerStore, BaseSlavedStore):
     def __init__(
         self,
         database: DatabasePool,

synapse/replication/slave/storage/directory.py

@@ -0,0 +1,21 @@
+# Copyright 2015, 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.storage.databases.main.directory import DirectoryWorkerStore
+
+from ._base import BaseSlavedStore
+
+
+class DirectoryStore(DirectoryWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/slave/storage/events.py

@@ -29,6 +29,8 @@ from synapse.storage.databases.main.stream import StreamWorkerStore
 from synapse.storage.databases.main.user_erasure_store import UserErasureWorkerStore
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 
+from ._base import BaseSlavedStore
+
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
@@ -54,6 +56,7 @@ class SlavedEventStore(
     EventsWorkerStore,
     UserErasureWorkerStore,
     RelationsWorkerStore,
+    BaseSlavedStore,
 ):
     def __init__(
         self,

synapse/replication/slave/storage/filtering.py

@@ -14,15 +14,16 @@
 
 from typing import TYPE_CHECKING
 
-from synapse.storage._base import SQLBaseStore
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
 from synapse.storage.databases.main.filtering import FilteringStore
 
+from ._base import BaseSlavedStore
+
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 
-class SlavedFilteringStore(SQLBaseStore):
+class SlavedFilteringStore(BaseSlavedStore):
     def __init__(
         self,
         database: DatabasePool,

synapse/replication/slave/storage/profile.py

@@ -0,0 +1,20 @@
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.storage.databases.main.profile import ProfileWorkerStore
+
+
+class SlavedProfileStore(ProfileWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/slave/storage/pushers.py

@@ -18,13 +18,14 @@ from synapse.replication.tcp.streams import PushersStream
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
 from synapse.storage.databases.main.pusher import PusherWorkerStore
 
+from ._base import BaseSlavedStore
 from ._slaved_id_tracker import SlavedIdTracker
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 
-class SlavedPusherStore(PusherWorkerStore):
+class SlavedPusherStore(PusherWorkerStore, BaseSlavedStore):
     def __init__(
         self,
         database: DatabasePool,

synapse/replication/slave/storage/receipts.py

@@ -0,0 +1,22 @@
+# Copyright 2016 OpenMarket Ltd
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
+
+from ._base import BaseSlavedStore
+
+
+class SlavedReceiptsStore(ReceiptsWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/slave/storage/registration.py

@@ -0,0 +1,21 @@
+# Copyright 2015, 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.storage.databases.main.registration import RegistrationWorkerStore
+
+from ._base import BaseSlavedStore
+
+
+class SlavedRegistrationStore(RegistrationWorkerStore, BaseSlavedStore):
+    pass

synapse/replication/tcp/client.py

@@ -21,7 +21,7 @@ from twisted.internet.interfaces import IAddress, IConnector
 from twisted.internet.protocol import ReconnectingClientFactory
 from twisted.python.failure import Failure
 
-from synapse.api.constants import EventTypes, Membership, ReceiptTypes
+from synapse.api.constants import EventTypes, ReceiptTypes
 from synapse.federation import send_queue
 from synapse.federation.sender import FederationSender
 from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
@@ -219,21 +219,6 @@ class ReplicationDataHandler:
                     membership=row.data.membership,
                 )
 
-                # If this event is a join, make a note of it so we have an accurate
-                # cross-worker room rate limit.
-                # TODO: Erik said we should exclude rows that came from ex_outliers
-                #  here, but I don't see how we can determine that. I guess we could
-                #  add a flag to row.data?
-                if (
-                    row.data.type == EventTypes.Member
-                    and row.data.membership == Membership.JOIN
-                    and not row.data.outlier
-                ):
-                    # TODO retrieve the previous state, and exclude join -> join transitions
-                    self.notifier.notify_user_joined_room(
-                        row.data.event_id, row.data.room_id
-                    )
-
         await self._presence_handler.process_replication_rows(
             stream_name, instance_name, token, rows
         )

synapse/replication/tcp/streams/events.py

@@ -98,7 +98,6 @@ class EventsStreamEventRow(BaseEventsStreamRow):
     relates_to: Optional[str]
     membership: Optional[str]
     rejected: bool
-    outlier: bool
 
 
 @attr.s(slots=True, frozen=True, auto_attribs=True)