Newsfile

Cap the top stream ID when fetching changed devices
2024-05-15 15:14:34 +01:00 · 2024-05-15 15:14:21 +01:00 · 2024-05-15 15:11:50 +01:00 · 2024-05-14 15:21:42 +01:00 · 2024-05-14 15:08:46 +01:00 · 2024-05-14 14:39:11 +01:00
34 changed files with 495 additions and 204 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,9 @@
+Synapse 1.107.0 (2024-05-14)
+============================
+
+No significant changes since 1.107.0rc1.
+
+
 # Synapse 1.107.0rc1 (2024-05-07)

 ### Features
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -485,18 +485,18 @@ checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"

 [[package]]
 name = "serde"
-version = "1.0.200"
+version = "1.0.201"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ddc6f9cc94d67c0e21aaf7eda3a010fd3af78ebf6e096aa6e2e13c79749cce4f"
+checksum = "780f1cebed1629e4753a1a38a3c72d30b97ec044f0aef68cb26650a3c5cf363c"
 dependencies = [
 "serde_derive",
 ]

 [[package]]
 name = "serde_derive"
-version = "1.0.200"
+version = "1.0.201"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "856f046b9400cee3c8c94ed572ecdb752444c24528c035cd35882aad6f492bcb"
+checksum = "c5e405930b9796f1c00bee880d03fc7e0bb4b9a11afc776885ffe84320da2865"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -505,9 +505,9 @@ dependencies = [

 [[package]]
 name = "serde_json"
-version = "1.0.116"
+version = "1.0.117"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3e17db7126d17feb94eb3fad46bf1a96b034e8aacbc2e775fe81505f8b0b2813"
+checksum = "455182ea6142b14f93f4bc5320a2b31c1f266b66a4a5c858b013302a5d8cbfc3"
 dependencies = [
 "itoa",
 "ryu",
--- a/changelog.d/16848.feature
+++ b/changelog.d/16848.feature
@@ -0,0 +1 @@
+Add a feature that allows clients to query the configured federation whitelist. Disabled by default.
--- a/changelog.d/17098.feature
+++ b/changelog.d/17098.feature
@@ -0,0 +1 @@
+Add the ability to allow numeric user IDs with a specific prefix when in the CAS flow. Contributed by Aurélien Grimpard.
--- a/changelog.d/17166.misc
+++ b/changelog.d/17166.misc
@@ -0,0 +1 @@
+Fixes linter errors found in PR #17147.
--- a/changelog.d/17170.misc
+++ b/changelog.d/17170.misc
@@ -0,0 +1 @@
+Bump black from 24.2.0 to 24.4.2.
--- a/changelog.d/17171.doc
+++ b/changelog.d/17171.doc
@@ -0,0 +1 @@
+Update the Admin FAQ with the current libjemalloc version for latest Debian stable. Additionally update the name of the "push_rules" stream in the Workers documentation.
--- a/changelog.d/17177.bugfix
+++ b/changelog.d/17177.bugfix
@@ -0,0 +1 @@
+Fix bug where disabling room publication prevented public rooms being created on workers.
--- a/changelog.d/17184.bugfix
+++ b/changelog.d/17184.bugfix
@@ -0,0 +1 @@
+Fix bug where disabling room publication prevented public rooms being created on workers.
--- a/changelog.d/17186.misc
+++ b/changelog.d/17186.misc
@@ -0,0 +1 @@
+Cache literal sync filter validation for performance.
--- a/changelog.d/17191.misc
+++ b/changelog.d/17191.misc
@@ -0,0 +1 @@
+Improve performance of calculating device lists changes in `/sync`.
--- a/changelog.d/17192.misc
+++ b/changelog.d/17192.misc
@@ -0,0 +1 @@
+Improve performance by fixing a reactor pause.
--- a/changelog.d/17197.bugfix
+++ b/changelog.d/17197.bugfix
@@ -0,0 +1 @@
+Fix bug where we sent down redundant device list changes down `/sync`.
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.107.0) stable; urgency=medium
+
+  * New Synapse release 1.107.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 14 May 2024 14:15:34 +0100
+
 matrix-synapse-py3 (1.107.0~rc1) stable; urgency=medium

  * New Synapse release 1.107.0rc1.
--- a/docs/usage/administration/admin_faq.md
+++ b/docs/usage/administration/admin_faq.md
@@ -250,10 +250,10 @@ Using [libjemalloc](https://jemalloc.net) can also yield a significant
 improvement in overall memory use, and especially in terms of giving back
 RAM to the OS. To use it, the library must simply be put in the
 LD_PRELOAD environment variable when launching Synapse. On Debian, this
-can be done by installing the `libjemalloc1` package and adding this
+can be done by installing the `libjemalloc2` package and adding this
 line to `/etc/default/matrix-synapse`:

-    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
+    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

 This made a significant difference on Python 2.7 - it's unclear how
 much of an improvement it provides on Python 3.x.
--- a/docs/usage/configuration/config_documentation.md
+++ b/docs/usage/configuration/config_documentation.md
@@ -1232,6 +1232,31 @@ federation_domain_whitelist:
  - syd.example.com
 ```
 ---
+### `federation_whitelist_endpoint_enabled`
+
+Enables an endpoint for fetching the federation whitelist config.
+
+The request method and path is `GET /_synapse/client/config/federation_whitelist`, and the
+response format is:
+
+```json
+{
+    "whitelist_enabled": true,  // Whether the federation whitelist is being enforced
+    "whitelist": [  // Which server names are allowed by the whitelist
+        "example.com"
+    ]
+}
+```
+
+If `whitelist_enabled` is `false` then the server is permitted to federate with all others.
+
+The endpoint requires authentication.
+
+Example configuration:
+```yaml
+federation_whitelist_endpoint_enabled: true
+```
+---
 ### `federation_metrics_domains`

 Report prometheus metrics on the age of PDUs being sent to and received from
@@ -3533,6 +3558,15 @@ Has the following sub-options:
   users. This allows the CAS SSO flow to be limited to sign in only, rather than
   automatically registering users that have a valid SSO login but do not have
   a pre-registered account. Defaults to true.
+* `allow_numeric_ids`: set to 'true' allow numeric user IDs (default false).
+   This allows CAS SSO flow to provide user IDs composed of numbers only.
+   These identifiers will be prefixed by the letter "u" by default.
+   The prefix can be configured using the "numeric_ids_prefix" option.
+   Be careful to choose the prefix correctly to avoid any possible conflicts
+   (e.g. user 1234 becomes u1234 when a user u1234 already exists).
+* `numeric_ids_prefix`: the prefix you wish to add in front of a numeric user ID
+   when the "allow_numeric_ids" option is set to "true".
+   By default, the prefix is the letter "u" and only alphanumeric characters are allowed.

   *Added in Synapse 1.93.0.*

@@ -3547,6 +3581,8 @@ cas_config:
    userGroup: "staff"
    department: None
  enable_registration: true
+  allow_numeric_ids: true
+  numeric_ids_prefix: "numericuser"
 ```
 ---
 ### `sso`
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -535,7 +535,7 @@ the stream writer for the `presence` stream:
 ##### The `push_rules` stream

 The following endpoints should be routed directly to the worker configured as
-the stream writer for the `push` stream:
+the stream writer for the `push_rules` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/

--- a/poetry.lock
+++ b/poetry.lock
@@ -107,33 +107,33 @@ typecheck = ["mypy"]

 [[package]]
 name = "black"
-version = "24.2.0"
+version = "24.4.2"
 description = "The uncompromising code formatter."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "black-24.2.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6981eae48b3b33399c8757036c7f5d48a535b962a7c2310d19361edeef64ce29"},
-    {file = "black-24.2.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:d533d5e3259720fdbc1b37444491b024003e012c5173f7d06825a77508085430"},
-    {file = "black-24.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:61a0391772490ddfb8a693c067df1ef5227257e72b0e4108482b8d41b5aee13f"},
-    {file = "black-24.2.0-cp310-cp310-win_amd64.whl", hash = "sha256:992e451b04667116680cb88f63449267c13e1ad134f30087dec8527242e9862a"},
-    {file = "black-24.2.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:163baf4ef40e6897a2a9b83890e59141cc8c2a98f2dda5080dc15c00ee1e62cd"},
-    {file = "black-24.2.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e37c99f89929af50ffaf912454b3e3b47fd64109659026b678c091a4cd450fb2"},
-    {file = "black-24.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f9de21bafcba9683853f6c96c2d515e364aee631b178eaa5145fc1c61a3cc92"},
-    {file = "black-24.2.0-cp311-cp311-win_amd64.whl", hash = "sha256:9db528bccb9e8e20c08e716b3b09c6bdd64da0dd129b11e160bf082d4642ac23"},
-    {file = "black-24.2.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:d84f29eb3ee44859052073b7636533ec995bd0f64e2fb43aeceefc70090e752b"},
-    {file = "black-24.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1e08fb9a15c914b81dd734ddd7fb10513016e5ce7e6704bdd5e1251ceee51ac9"},
-    {file = "black-24.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:810d445ae6069ce64030c78ff6127cd9cd178a9ac3361435708b907d8a04c693"},
-    {file = "black-24.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:ba15742a13de85e9b8f3239c8f807723991fbfae24bad92d34a2b12e81904982"},
-    {file = "black-24.2.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:7e53a8c630f71db01b28cd9602a1ada68c937cbf2c333e6ed041390d6968faf4"},
-    {file = "black-24.2.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:93601c2deb321b4bad8f95df408e3fb3943d85012dddb6121336b8e24a0d1218"},
-    {file = "black-24.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a0057f800de6acc4407fe75bb147b0c2b5cbb7c3ed110d3e5999cd01184d53b0"},
-    {file = "black-24.2.0-cp38-cp38-win_amd64.whl", hash = "sha256:faf2ee02e6612577ba0181f4347bcbcf591eb122f7841ae5ba233d12c39dcb4d"},
-    {file = "black-24.2.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:057c3dc602eaa6fdc451069bd027a1b2635028b575a6c3acfd63193ced20d9c8"},
-    {file = "black-24.2.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:08654d0797e65f2423f850fc8e16a0ce50925f9337fb4a4a176a7aa4026e63f8"},
-    {file = "black-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca610d29415ee1a30a3f30fab7a8f4144e9d34c89a235d81292a1edb2b55f540"},
-    {file = "black-24.2.0-cp39-cp39-win_amd64.whl", hash = "sha256:4dd76e9468d5536abd40ffbc7a247f83b2324f0c050556d9c371c2b9a9a95e31"},
-    {file = "black-24.2.0-py3-none-any.whl", hash = "sha256:e8a6ae970537e67830776488bca52000eaa37fa63b9988e8c487458d9cd5ace6"},
-    {file = "black-24.2.0.tar.gz", hash = "sha256:bce4f25c27c3435e4dace4815bcb2008b87e167e3bf4ee47ccdc5ce906eb4894"},
+    {file = "black-24.4.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:dd1b5a14e417189db4c7b64a6540f31730713d173f0b63e55fabd52d61d8fdce"},
+    {file = "black-24.4.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e537d281831ad0e71007dcdcbe50a71470b978c453fa41ce77186bbe0ed6021"},
+    {file = "black-24.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eaea3008c281f1038edb473c1aa8ed8143a5535ff18f978a318f10302b254063"},
+    {file = "black-24.4.2-cp310-cp310-win_amd64.whl", hash = "sha256:7768a0dbf16a39aa5e9a3ded568bb545c8c2727396d063bbaf847df05b08cd96"},
+    {file = "black-24.4.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:257d724c2c9b1660f353b36c802ccece186a30accc7742c176d29c146df6e474"},
+    {file = "black-24.4.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bdde6f877a18f24844e381d45e9947a49e97933573ac9d4345399be37621e26c"},
+    {file = "black-24.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e151054aa00bad1f4e1f04919542885f89f5f7d086b8a59e5000e6c616896ffb"},
+    {file = "black-24.4.2-cp311-cp311-win_amd64.whl", hash = "sha256:7e122b1c4fb252fd85df3ca93578732b4749d9be076593076ef4d07a0233c3e1"},
+    {file = "black-24.4.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:accf49e151c8ed2c0cdc528691838afd217c50412534e876a19270fea1e28e2d"},
+    {file = "black-24.4.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:88c57dc656038f1ab9f92b3eb5335ee9b021412feaa46330d5eba4e51fe49b04"},
+    {file = "black-24.4.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be8bef99eb46d5021bf053114442914baeb3649a89dc5f3a555c88737e5e98fc"},
+    {file = "black-24.4.2-cp312-cp312-win_amd64.whl", hash = "sha256:415e686e87dbbe6f4cd5ef0fbf764af7b89f9057b97c908742b6008cc554b9c0"},
+    {file = "black-24.4.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:bf10f7310db693bb62692609b397e8d67257c55f949abde4c67f9cc574492cc7"},
+    {file = "black-24.4.2-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:98e123f1d5cfd42f886624d84464f7756f60ff6eab89ae845210631714f6db94"},
+    {file = "black-24.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:48a85f2cb5e6799a9ef05347b476cce6c182d6c71ee36925a6c194d074336ef8"},
+    {file = "black-24.4.2-cp38-cp38-win_amd64.whl", hash = "sha256:b1530ae42e9d6d5b670a34db49a94115a64596bc77710b1d05e9801e62ca0a7c"},
+    {file = "black-24.4.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:37aae07b029fa0174d39daf02748b379399b909652a806e5708199bd93899da1"},
+    {file = "black-24.4.2-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:da33a1a5e49c4122ccdfd56cd021ff1ebc4a1ec4e2d01594fef9b6f267a9e741"},
+    {file = "black-24.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ef703f83fc32e131e9bcc0a5094cfe85599e7109f896fe8bc96cc402f3eb4b6e"},
+    {file = "black-24.4.2-cp39-cp39-win_amd64.whl", hash = "sha256:b9176b9832e84308818a99a561e90aa479e73c523b3f77afd07913380ae2eab7"},
+    {file = "black-24.4.2-py3-none-any.whl", hash = "sha256:d36ed1124bb81b32f8614555b34cc4259c3fbc7eec17870e8ff8ded335b58d8c"},
+    {file = "black-24.4.2.tar.gz", hash = "sha256:c872b53057f000085da66a19c55d68f6f8ddcac2642392ad3a355878406fbd4d"},
 ]

 [package.dependencies]
@@ -403,43 +403,43 @@ files = [

 [[package]]
 name = "cryptography"
-version = "42.0.5"
+version = "42.0.7"
 description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "cryptography-42.0.5-cp37-abi3-macosx_10_12_universal2.whl", hash = "sha256:a30596bae9403a342c978fb47d9b0ee277699fa53bbafad14706af51fe543d16"},
-    {file = "cryptography-42.0.5-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:b7ffe927ee6531c78f81aa17e684e2ff617daeba7f189f911065b2ea2d526dec"},
-    {file = "cryptography-42.0.5-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2424ff4c4ac7f6b8177b53c17ed5d8fa74ae5955656867f5a8affaca36a27abb"},
-    {file = "cryptography-42.0.5-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:329906dcc7b20ff3cad13c069a78124ed8247adcac44b10bea1130e36caae0b4"},
-    {file = "cryptography-42.0.5-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:b03c2ae5d2f0fc05f9a2c0c997e1bc18c8229f392234e8a0194f202169ccd278"},
-    {file = "cryptography-42.0.5-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:f8837fe1d6ac4a8052a9a8ddab256bc006242696f03368a4009be7ee3075cdb7"},
-    {file = "cryptography-42.0.5-cp37-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:0270572b8bd2c833c3981724b8ee9747b3ec96f699a9665470018594301439ee"},
-    {file = "cryptography-42.0.5-cp37-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:b8cac287fafc4ad485b8a9b67d0ee80c66bf3574f655d3b97ef2e1082360faf1"},
-    {file = "cryptography-42.0.5-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:16a48c23a62a2f4a285699dba2e4ff2d1cff3115b9df052cdd976a18856d8e3d"},
-    {file = "cryptography-42.0.5-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2bce03af1ce5a5567ab89bd90d11e7bbdff56b8af3acbbec1faded8f44cb06da"},
-    {file = "cryptography-42.0.5-cp37-abi3-win32.whl", hash = "sha256:b6cd2203306b63e41acdf39aa93b86fb566049aeb6dc489b70e34bcd07adca74"},
-    {file = "cryptography-42.0.5-cp37-abi3-win_amd64.whl", hash = "sha256:98d8dc6d012b82287f2c3d26ce1d2dd130ec200c8679b6213b3c73c08b2b7940"},
-    {file = "cryptography-42.0.5-cp39-abi3-macosx_10_12_universal2.whl", hash = "sha256:5e6275c09d2badf57aea3afa80d975444f4be8d3bc58f7f80d2a484c6f9485c8"},
-    {file = "cryptography-42.0.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e4985a790f921508f36f81831817cbc03b102d643b5fcb81cd33df3fa291a1a1"},
-    {file = "cryptography-42.0.5-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7cde5f38e614f55e28d831754e8a3bacf9ace5d1566235e39d91b35502d6936e"},
-    {file = "cryptography-42.0.5-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:7367d7b2eca6513681127ebad53b2582911d1736dc2ffc19f2c3ae49997496bc"},
-    {file = "cryptography-42.0.5-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:cd2030f6650c089aeb304cf093f3244d34745ce0cfcc39f20c6fbfe030102e2a"},
-    {file = "cryptography-42.0.5-cp39-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:a2913c5375154b6ef2e91c10b5720ea6e21007412f6437504ffea2109b5a33d7"},
-    {file = "cryptography-42.0.5-cp39-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:c41fb5e6a5fe9ebcd58ca3abfeb51dffb5d83d6775405305bfa8715b76521922"},
-    {file = "cryptography-42.0.5-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:3eaafe47ec0d0ffcc9349e1708be2aaea4c6dd4978d76bf6eb0cb2c13636c6fc"},
-    {file = "cryptography-42.0.5-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1b95b98b0d2af784078fa69f637135e3c317091b615cd0905f8b8a087e86fa30"},
-    {file = "cryptography-42.0.5-cp39-abi3-win32.whl", hash = "sha256:1f71c10d1e88467126f0efd484bd44bca5e14c664ec2ede64c32f20875c0d413"},
-    {file = "cryptography-42.0.5-cp39-abi3-win_amd64.whl", hash = "sha256:a011a644f6d7d03736214d38832e030d8268bcff4a41f728e6030325fea3e400"},
-    {file = "cryptography-42.0.5-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:9481ffe3cf013b71b2428b905c4f7a9a4f76ec03065b05ff499bb5682a8d9ad8"},
-    {file = "cryptography-42.0.5-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:ba334e6e4b1d92442b75ddacc615c5476d4ad55cc29b15d590cc6b86efa487e2"},
-    {file = "cryptography-42.0.5-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:ba3e4a42397c25b7ff88cdec6e2a16c2be18720f317506ee25210f6d31925f9c"},
-    {file = "cryptography-42.0.5-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:111a0d8553afcf8eb02a4fea6ca4f59d48ddb34497aa8706a6cf536f1a5ec576"},
-    {file = "cryptography-42.0.5-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:cd65d75953847815962c84a4654a84850b2bb4aed3f26fadcc1c13892e1e29f6"},
-    {file = "cryptography-42.0.5-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:e807b3188f9eb0eaa7bbb579b462c5ace579f1cedb28107ce8b48a9f7ad3679e"},
-    {file = "cryptography-42.0.5-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:f12764b8fffc7a123f641d7d049d382b73f96a34117e0b637b80643169cec8ac"},
-    {file = "cryptography-42.0.5-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:37dd623507659e08be98eec89323469e8c7b4c1407c85112634ae3dbdb926fdd"},
-    {file = "cryptography-42.0.5.tar.gz", hash = "sha256:6fe07eec95dfd477eb9530aef5bead34fec819b3aaf6c5bd6d20565da607bfe1"},
+    {file = "cryptography-42.0.7-cp37-abi3-macosx_10_12_universal2.whl", hash = "sha256:a987f840718078212fdf4504d0fd4c6effe34a7e4740378e59d47696e8dfb477"},
+    {file = "cryptography-42.0.7-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:bd13b5e9b543532453de08bcdc3cc7cebec6f9883e886fd20a92f26940fd3e7a"},
+    {file = "cryptography-42.0.7-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a79165431551042cc9d1d90e6145d5d0d3ab0f2d66326c201d9b0e7f5bf43604"},
+    {file = "cryptography-42.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a47787a5e3649008a1102d3df55424e86606c9bae6fb77ac59afe06d234605f8"},
+    {file = "cryptography-42.0.7-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:02c0eee2d7133bdbbc5e24441258d5d2244beb31da5ed19fbb80315f4bbbff55"},
+    {file = "cryptography-42.0.7-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:5e44507bf8d14b36b8389b226665d597bc0f18ea035d75b4e53c7b1ea84583cc"},
+    {file = "cryptography-42.0.7-cp37-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:7f8b25fa616d8b846aef64b15c606bb0828dbc35faf90566eb139aa9cff67af2"},
+    {file = "cryptography-42.0.7-cp37-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:93a3209f6bb2b33e725ed08ee0991b92976dfdcf4e8b38646540674fc7508e13"},
+    {file = "cryptography-42.0.7-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:e6b8f1881dac458c34778d0a424ae5769de30544fc678eac51c1c8bb2183e9da"},
+    {file = "cryptography-42.0.7-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3de9a45d3b2b7d8088c3fbf1ed4395dfeff79d07842217b38df14ef09ce1d8d7"},
+    {file = "cryptography-42.0.7-cp37-abi3-win32.whl", hash = "sha256:789caea816c6704f63f6241a519bfa347f72fbd67ba28d04636b7c6b7da94b0b"},
+    {file = "cryptography-42.0.7-cp37-abi3-win_amd64.whl", hash = "sha256:8cb8ce7c3347fcf9446f201dc30e2d5a3c898d009126010cbd1f443f28b52678"},
+    {file = "cryptography-42.0.7-cp39-abi3-macosx_10_12_universal2.whl", hash = "sha256:a3a5ac8b56fe37f3125e5b72b61dcde43283e5370827f5233893d461b7360cd4"},
+    {file = "cryptography-42.0.7-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:779245e13b9a6638df14641d029add5dc17edbef6ec915688f3acb9e720a5858"},
+    {file = "cryptography-42.0.7-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0d563795db98b4cd57742a78a288cdbdc9daedac29f2239793071fe114f13785"},
+    {file = "cryptography-42.0.7-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:31adb7d06fe4383226c3e963471f6837742889b3c4caa55aac20ad951bc8ffda"},
+    {file = "cryptography-42.0.7-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:efd0bf5205240182e0f13bcaea41be4fdf5c22c5129fc7ced4a0282ac86998c9"},
+    {file = "cryptography-42.0.7-cp39-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:a9bc127cdc4ecf87a5ea22a2556cab6c7eda2923f84e4f3cc588e8470ce4e42e"},
+    {file = "cryptography-42.0.7-cp39-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:3577d029bc3f4827dd5bf8bf7710cac13527b470bbf1820a3f394adb38ed7d5f"},
+    {file = "cryptography-42.0.7-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2e47577f9b18723fa294b0ea9a17d5e53a227867a0a4904a1a076d1646d45ca1"},
+    {file = "cryptography-42.0.7-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1a58839984d9cb34c855197043eaae2c187d930ca6d644612843b4fe8513c886"},
+    {file = "cryptography-42.0.7-cp39-abi3-win32.whl", hash = "sha256:e6b79d0adb01aae87e8a44c2b64bc3f3fe59515280e00fb6d57a7267a2583cda"},
+    {file = "cryptography-42.0.7-cp39-abi3-win_amd64.whl", hash = "sha256:16268d46086bb8ad5bf0a2b5544d8a9ed87a0e33f5e77dd3c3301e63d941a83b"},
+    {file = "cryptography-42.0.7-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:2954fccea107026512b15afb4aa664a5640cd0af630e2ee3962f2602693f0c82"},
+    {file = "cryptography-42.0.7-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:362e7197754c231797ec45ee081f3088a27a47c6c01eff2ac83f60f85a50fe60"},
+    {file = "cryptography-42.0.7-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:4f698edacf9c9e0371112792558d2f705b5645076cc0aaae02f816a0171770fd"},
+    {file = "cryptography-42.0.7-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:5482e789294854c28237bba77c4c83be698be740e31a3ae5e879ee5444166582"},
+    {file = "cryptography-42.0.7-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:e9b2a6309f14c0497f348d08a065d52f3020656f675819fc405fb63bbcd26562"},
+    {file = "cryptography-42.0.7-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:d8e3098721b84392ee45af2dd554c947c32cc52f862b6a3ae982dbb90f577f14"},
+    {file = "cryptography-42.0.7-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c65f96dad14f8528a447414125e1fc8feb2ad5a272b8f68477abbcc1ea7d94b9"},
+    {file = "cryptography-42.0.7-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:36017400817987670037fbb0324d71489b6ead6231c9604f8fc1f7d008087c68"},
+    {file = "cryptography-42.0.7.tar.gz", hash = "sha256:ecbfbc00bf55888edda9868a4cf927205de8499e7fabe6c050322298382953f2"},
 ]

 [package.dependencies]
@@ -524,20 +524,21 @@ smmap = ">=3.0.1,<6"

 [[package]]
 name = "gitpython"
-version = "3.1.41"
+version = "3.1.43"
 description = "GitPython is a Python library used to interact with Git repositories"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "GitPython-3.1.41-py3-none-any.whl", hash = "sha256:c36b6634d069b3f719610175020a9aed919421c87552185b085e04fbbdb10b7c"},
-    {file = "GitPython-3.1.41.tar.gz", hash = "sha256:ed66e624884f76df22c8e16066d567aaa5a37d5b5fa19db2c6df6f7156db9048"},
+    {file = "GitPython-3.1.43-py3-none-any.whl", hash = "sha256:eec7ec56b92aad751f9912a73404bc02ba212a23adb2c7098ee668417051a1ff"},
+    {file = "GitPython-3.1.43.tar.gz", hash = "sha256:35f314a9f878467f5453cc1fee295c3e18e52f1b99f10f6cf5b1682e968a9e7c"},
 ]

 [package.dependencies]
 gitdb = ">=4.0.1,<5"

 [package.extras]
-test = ["black", "coverage[toml]", "ddt (>=1.1.1,!=1.4.3)", "mock", "mypy", "pre-commit", "pytest (>=7.3.1)", "pytest-cov", "pytest-instafail", "pytest-mock", "pytest-sugar", "sumtypes"]
+doc = ["sphinx (==4.3.2)", "sphinx-autodoc-typehints", "sphinx-rtd-theme", "sphinxcontrib-applehelp (>=1.0.2,<=1.0.4)", "sphinxcontrib-devhelp (==1.0.2)", "sphinxcontrib-htmlhelp (>=2.0.0,<=2.0.1)", "sphinxcontrib-qthelp (==1.0.3)", "sphinxcontrib-serializinghtml (==1.1.5)"]
+test = ["coverage[toml]", "ddt (>=1.1.1,!=1.4.3)", "mock", "mypy", "pre-commit", "pytest (>=7.3.1)", "pytest-cov", "pytest-instafail", "pytest-mock", "pytest-sugar", "typing-extensions"]

 [[package]]
 name = "hiredis"
@@ -782,13 +783,13 @@ files = [

 [[package]]
 name = "immutabledict"
-version = "4.1.0"
+version = "4.2.0"
 description = "Immutable wrapper around dictionaries (a fork of frozendict)"
 optional = false
 python-versions = ">=3.8,<4.0"
 files = [
-    {file = "immutabledict-4.1.0-py3-none-any.whl", hash = "sha256:c176e99aa90aedb81716ad35218bb2055d049b549626db4523dbe011cf2f32ac"},
-    {file = "immutabledict-4.1.0.tar.gz", hash = "sha256:93d100ccd2cd09a1fd3f136b9328c6e59529ba341de8bb499437f6819159fe8a"},
+    {file = "immutabledict-4.2.0-py3-none-any.whl", hash = "sha256:d728b2c2410d698d95e6200237feb50a695584d20289ad3379a439aa3d90baba"},
+    {file = "immutabledict-4.2.0.tar.gz", hash = "sha256:e003fd81aad2377a5a758bf7e1086cf3b70b63e9a5cc2f46bce8d0a2b4727c5f"},
 ]

 [[package]]
@@ -2398,26 +2399,28 @@ doc = ["Sphinx", "sphinx-rtd-theme"]

 [[package]]
 name = "sentry-sdk"
-version = "1.40.3"
+version = "2.1.1"
 description = "Python client for Sentry (https://sentry.io)"
 optional = true
-python-versions = "*"
+python-versions = ">=3.6"
 files = [
-    {file = "sentry-sdk-1.40.3.tar.gz", hash = "sha256:3c2b027979bb400cd65a47970e64f8cef8acda86b288a27f42a98692505086cd"},
-    {file = "sentry_sdk-1.40.3-py2.py3-none-any.whl", hash = "sha256:73383f28311ae55602bb6cc3b013830811135ba5521e41333a6e68f269413502"},
+    {file = "sentry_sdk-2.1.1-py2.py3-none-any.whl", hash = "sha256:99aeb78fb76771513bd3b2829d12613130152620768d00cd3e45ac00cb17950f"},
+    {file = "sentry_sdk-2.1.1.tar.gz", hash = "sha256:95d8c0bb41c8b0bc37ab202c2c4a295bb84398ee05f4cdce55051cd75b926ec1"},
 ]

 [package.dependencies]
 certifi = "*"
-urllib3 = {version = ">=1.26.11", markers = "python_version >= \"3.6\""}
+urllib3 = ">=1.26.11"

 [package.extras]
 aiohttp = ["aiohttp (>=3.5)"]
+anthropic = ["anthropic (>=0.16)"]
 arq = ["arq (>=0.23)"]
 asyncpg = ["asyncpg (>=0.23)"]
 beam = ["apache-beam (>=2.12)"]
 bottle = ["bottle (>=0.12.13)"]
 celery = ["celery (>=3)"]
+celery-redbeat = ["celery-redbeat (>=2)"]
 chalice = ["chalice (>=1.16.0)"]
 clickhouse-driver = ["clickhouse-driver (>=0.2.0)"]
 django = ["django (>=1.8)"]
@@ -2427,7 +2430,10 @@ flask = ["blinker (>=1.1)", "flask (>=0.11)", "markupsafe"]
 grpcio = ["grpcio (>=1.21.1)"]
 httpx = ["httpx (>=0.16.0)"]
 huey = ["huey (>=2)"]
+huggingface-hub = ["huggingface-hub (>=0.22)"]
+langchain = ["langchain (>=0.0.210)"]
 loguru = ["loguru (>=0.5)"]
+openai = ["openai (>=1.0.0)", "tiktoken (>=0.3.0)"]
 opentelemetry = ["opentelemetry-distro (>=0.35b0)"]
 opentelemetry-experimental = ["opentelemetry-distro (>=0.40b0,<1.0)", "opentelemetry-instrumentation-aiohttp-client (>=0.40b0,<1.0)", "opentelemetry-instrumentation-django (>=0.40b0,<1.0)", "opentelemetry-instrumentation-fastapi (>=0.40b0,<1.0)", "opentelemetry-instrumentation-flask (>=0.40b0,<1.0)", "opentelemetry-instrumentation-requests (>=0.40b0,<1.0)", "opentelemetry-instrumentation-sqlite3 (>=0.40b0,<1.0)", "opentelemetry-instrumentation-urllib (>=0.40b0,<1.0)"]
 pure-eval = ["asttokens", "executing", "pure-eval"]
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -96,7 +96,7 @@ module-name = "synapse.synapse_rust"

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.107.0rc1"
+version = "1.107.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"
--- a/synapse/config/cas.py
+++ b/synapse/config/cas.py
@@ -66,6 +66,17 @@ class CasConfig(Config):

            self.cas_enable_registration = cas_config.get("enable_registration", True)

+            self.cas_allow_numeric_ids = cas_config.get("allow_numeric_ids")
+            self.cas_numeric_ids_prefix = cas_config.get("numeric_ids_prefix")
+            if (
+                self.cas_numeric_ids_prefix is not None
+                and self.cas_numeric_ids_prefix.isalnum() is False
+            ):
+                raise ConfigError(
+                    "Only alphanumeric characters are allowed for numeric IDs prefix",
+                    ("cas_config", "numeric_ids_prefix"),
+                )
+
            self.idp_name = cas_config.get("idp_name", "CAS")
            self.idp_icon = cas_config.get("idp_icon")
            self.idp_brand = cas_config.get("idp_brand")
@@ -77,6 +88,8 @@ class CasConfig(Config):
            self.cas_displayname_attribute = None
            self.cas_required_attributes = []
            self.cas_enable_registration = False
+            self.cas_allow_numeric_ids = False
+            self.cas_numeric_ids_prefix = "u"


 # CAS uses a legacy required attributes mapping, not the one provided by
--- a/synapse/config/federation.py
+++ b/synapse/config/federation.py
@@ -42,6 +42,10 @@ class FederationConfig(Config):
            for domain in federation_domain_whitelist:
                self.federation_domain_whitelist[domain] = True

+        self.federation_whitelist_endpoint_enabled = config.get(
+            "federation_whitelist_endpoint_enabled", False
+        )
+
        federation_metrics_domains = config.get("federation_metrics_domains") or []
        validate_config(
            _METRICS_FOR_DOMAINS_SCHEMA,
--- a/synapse/handlers/cas.py
+++ b/synapse/handlers/cas.py
@@ -78,6 +78,8 @@ class CasHandler:
        self._cas_displayname_attribute = hs.config.cas.cas_displayname_attribute
        self._cas_required_attributes = hs.config.cas.cas_required_attributes
        self._cas_enable_registration = hs.config.cas.cas_enable_registration
+        self._cas_allow_numeric_ids = hs.config.cas.cas_allow_numeric_ids
+        self._cas_numeric_ids_prefix = hs.config.cas.cas_numeric_ids_prefix

        self._http_client = hs.get_proxied_http_client()

@@ -188,6 +190,9 @@ class CasHandler:
        for child in root[0]:
            if child.tag.endswith("user"):
                user = child.text
+                # if numeric user IDs are allowed and username is numeric then we add the prefix so Synapse can handle it
+                if self._cas_allow_numeric_ids and user is not None and user.isdigit():
+                    user = f"{self._cas_numeric_ids_prefix}{user}"
            if child.tag.endswith("attributes"):
                for attribute in child:
                    # ElementTree library expands the namespace in
--- a/synapse/handlers/device.py
+++ b/synapse/handlers/device.py
@@ -159,20 +159,32 @@ class DeviceWorkerHandler:

    @cancellable
    async def get_device_changes_in_shared_rooms(
-        self, user_id: str, room_ids: StrCollection, from_token: StreamToken
+        self,
+        user_id: str,
+        room_ids: StrCollection,
+        from_token: StreamToken,
+        now_token: Optional[StreamToken] = None,
    ) -> Set[str]:
        """Get the set of users whose devices have changed who share a room with
        the given user.
        """
+        now_device_lists_key = self.store.get_device_stream_token()
+        if now_token:
+            now_device_lists_key = now_token.device_list_key
+
        changed_users = await self.store.get_device_list_changes_in_rooms(
-            room_ids, from_token.device_list_key
+            room_ids,
+            from_token.device_list_key,
+            now_device_lists_key,
        )

        if changed_users is not None:
            # We also check if the given user has changed their device. If
            # they're in no rooms then the above query won't include them.
            changed = await self.store.get_users_whose_devices_changed(
-                from_token.device_list_key, [user_id]
+                from_token.device_list_key,
+                [user_id],
+                to_key=now_device_lists_key,
            )
            changed_users.update(changed)
            return changed_users
@@ -190,7 +202,9 @@ class DeviceWorkerHandler:
        tracked_users.add(user_id)

        changed = await self.store.get_users_whose_devices_changed(
-            from_token.device_list_key, tracked_users
+            from_token.device_list_key,
+            tracked_users,
+            to_key=now_device_lists_key,
        )

        return changed
--- a/synapse/handlers/profile.py
+++ b/synapse/handlers/profile.py
@@ -20,7 +20,7 @@
 #
 import logging
 import random
-from typing import TYPE_CHECKING, Optional, Union
+from typing import TYPE_CHECKING, List, Optional, Union

 from synapse.api.errors import (
    AuthError,
@@ -64,8 +64,10 @@ class ProfileHandler:
        self.user_directory_handler = hs.get_user_directory_handler()
        self.request_ratelimiter = hs.get_request_ratelimiter()

-        self.max_avatar_size = hs.config.server.max_avatar_size
-        self.allowed_avatar_mimetypes = hs.config.server.allowed_avatar_mimetypes
+        self.max_avatar_size: Optional[int] = hs.config.server.max_avatar_size
+        self.allowed_avatar_mimetypes: Optional[List[str]] = (
+            hs.config.server.allowed_avatar_mimetypes
+        )

        self._is_mine_server_name = hs.is_mine_server_name

@@ -337,6 +339,12 @@ class ProfileHandler:
            return False

        if self.max_avatar_size:
+            if media_info.media_length is None:
+                logger.warning(
+                    "Forbidding avatar change to %s: unknown media size",
+                    mxc,
+                )
+                return False
            # Ensure avatar does not exceed max allowed avatar size
            if media_info.media_length > self.max_avatar_size:
                logger.warning(
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -1803,38 +1803,14 @@ class SyncHandler:

        # Step 1a, check for changes in devices of users we share a room
        # with
-        #
-        # We do this in two different ways depending on what we have cached.
-        # If we already have a list of all the user that have changed since
-        # the last sync then it's likely more efficient to compare the rooms
-        # they're in with the rooms the syncing user is in.
-        #
-        # If we don't have that info cached then we get all the users that
-        # share a room with our user and check if those users have changed.
-        cache_result = self.store.get_cached_device_list_changes(
-            since_token.device_list_key
-        )
-        if cache_result.hit:
-            changed_users = cache_result.entities
-
-            result = await self.store.get_rooms_for_users(changed_users)
-
-            for changed_user_id, entries in result.items():
-                # Check if the changed user shares any rooms with the user,
-                # or if the changed user is the syncing user (as we always
-                # want to include device list updates of their own devices).
-                if user_id == changed_user_id or any(
-                    rid in joined_rooms for rid in entries
-                ):
-                    users_that_have_changed.add(changed_user_id)
-        else:
-            users_that_have_changed = (
-                await self._device_handler.get_device_changes_in_shared_rooms(
-                    user_id,
-                    sync_result_builder.joined_room_ids,
-                    from_token=since_token,
-                )
+        users_that_have_changed = (
+            await self._device_handler.get_device_changes_in_shared_rooms(
+                user_id,
+                sync_result_builder.joined_room_ids,
+                from_token=since_token,
+                now_token=sync_result_builder.now_token,
            )
+        )

        # Step 1b, check for newly joined rooms
        for room_id in newly_joined_rooms:
--- a/synapse/replication/tcp/client.py
+++ b/synapse/replication/tcp/client.py
@@ -55,6 +55,7 @@ from synapse.replication.tcp.streams.partial_state import (
 )
 from synapse.types import PersistedEventPosition, ReadReceipt, StreamKeyType, UserID
 from synapse.util.async_helpers import Linearizer, timeout_deferred
+from synapse.util.iterutils import batch_iter
 from synapse.util.metrics import Measure

 if TYPE_CHECKING:
@@ -150,9 +151,15 @@ class ReplicationDataHandler:
                if row.entity.startswith("@") and not row.is_signature:
                    room_ids = await self.store.get_rooms_for_user(row.entity)
                    all_room_ids.update(room_ids)
-            self.notifier.on_new_event(
-                StreamKeyType.DEVICE_LIST, token, rooms=all_room_ids
-            )
+
+            # `all_room_ids` can be large, so let's wake up those streams in batches
+            for batched_room_ids in batch_iter(all_room_ids, 100):
+                self.notifier.on_new_event(
+                    StreamKeyType.DEVICE_LIST, token, rooms=batched_room_ids
+                )
+
+                # Yield to reactor so that we don't block.
+                await self._clock.sleep(0)
        elif stream_name == PushersStream.NAME:
            for row in rows:
                if row.deleted:
--- a/synapse/rest/client/sync.py
+++ b/synapse/rest/client/sync.py
@@ -47,6 +47,7 @@ from synapse.http.site import SynapseRequest
 from synapse.logging.opentracing import trace_with_opname
 from synapse.types import JsonDict, Requester, StreamToken
 from synapse.util import json_decoder
+from synapse.util.caches.lrucache import LruCache

 from ._base import client_patterns, set_timeline_upper_limit

@@ -110,6 +111,11 @@ class SyncRestServlet(RestServlet):
        self._msc2654_enabled = hs.config.experimental.msc2654_enabled
        self._msc3773_enabled = hs.config.experimental.msc3773_enabled

+        self._json_filter_cache: LruCache[str, bool] = LruCache(
+            max_size=1000,
+            cache_name="sync_valid_filter",
+        )
+
    async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
        # This will always be set by the time Twisted calls us.
        assert request.args is not None
@@ -177,7 +183,13 @@ class SyncRestServlet(RestServlet):
                filter_object = json_decoder.decode(filter_id)
            except Exception:
                raise SynapseError(400, "Invalid filter JSON", errcode=Codes.NOT_JSON)
-            self.filtering.check_valid_filter(filter_object)
+
+            # We cache the validation, as this can get quite expensive if people use
+            # a literal json blob as a query param.
+            if not self._json_filter_cache.get(filter_id):
+                self.filtering.check_valid_filter(filter_object)
+                self._json_filter_cache[filter_id] = True
+
            set_timeline_upper_limit(
                filter_object, self.hs.config.server.filter_timeline_limit
            )
--- a/synapse/rest/synapse/client/init.py
+++ b/synapse/rest/synapse/client/init.py
@@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Mapping

 from twisted.web.resource import Resource

+from synapse.rest.synapse.client.federation_whitelist import FederationWhitelistResource
 from synapse.rest.synapse.client.new_user_consent import NewUserConsentResource
 from synapse.rest.synapse.client.pick_idp import PickIdpResource
 from synapse.rest.synapse.client.pick_username import pick_username_resource
@@ -77,6 +78,9 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc
        # To be removed in Synapse v1.32.0.
        resources["/_matrix/saml2"] = res

+    if hs.config.federation.federation_whitelist_endpoint_enabled:
+        resources[FederationWhitelistResource.PATH] = FederationWhitelistResource(hs)
+
    if hs.config.experimental.msc4108_enabled:
        resources["/_synapse/client/rendezvous"] = MSC4108RendezvousSessionResource(hs)

--- a/synapse/rest/synapse/client/federation_whitelist.py
+++ b/synapse/rest/synapse/client/federation_whitelist.py
@@ -0,0 +1,66 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2024 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
+#
+
+import logging
+from typing import TYPE_CHECKING, Tuple
+
+from synapse.http.server import DirectServeJsonResource
+from synapse.http.site import SynapseRequest
+from synapse.types import JsonDict
+
+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
+logger = logging.getLogger(__name__)
+
+
+class FederationWhitelistResource(DirectServeJsonResource):
+    """Custom endpoint (disabled by default) to fetch the federation whitelist
+    config.
+
+    Only enabled if `federation_whitelist_endpoint_enabled` feature is enabled.
+
+    Response format:
+
+        {
+            "whitelist_enabled": true,  // Whether the federation whitelist is being enforced
+            "whitelist": [  // Which server names are allowed by the whitelist
+                "example.com"
+            ]
+        }
+    """
+
+    PATH = "/_synapse/client/v1/config/federation_whitelist"
+
+    def __init__(self, hs: "HomeServer"):
+        super().__init__()
+
+        self._federation_whitelist = hs.config.federation.federation_domain_whitelist
+
+        self._auth = hs.get_auth()
+
+    async def _async_render_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
+        await self._auth.get_user_by_req(request)
+
+        whitelist = []
+        if self._federation_whitelist:
+            # federation_whitelist is actually a dict, not a list
+            whitelist = list(self._federation_whitelist)
+
+        return_dict: JsonDict = {
+            "whitelist_enabled": self._federation_whitelist is not None,
+            "whitelist": whitelist,
+        }
+
+        return 200, return_dict
--- a/synapse/storage/databases/main/devices.py
+++ b/synapse/storage/databases/main/devices.py
@@ -70,10 +70,7 @@ from synapse.types import (
 from synapse.util import json_decoder, json_encoder
 from synapse.util.caches.descriptors import cached, cachedList
 from synapse.util.caches.lrucache import LruCache
-from synapse.util.caches.stream_change_cache import (
-    AllEntitiesChangedResult,
-    StreamChangeCache,
-)
+from synapse.util.caches.stream_change_cache import StreamChangeCache
 from synapse.util.cancellation import cancellable
 from synapse.util.iterutils import batch_iter
 from synapse.util.stringutils import shortstr
@@ -832,16 +829,6 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
        )
        return {device[0]: db_to_json(device[1]) for device in devices}

-    def get_cached_device_list_changes(
-        self,
-        from_key: int,
-    ) -> AllEntitiesChangedResult:
-        """Get set of users whose devices have changed since `from_key`, or None
-        if that information is not in our cache.
-        """
-
-        return self._device_list_stream_cache.get_all_entities_changed(from_key)
-
    @cancellable
    async def get_all_devices_changed(
        self,
@@ -1457,7 +1444,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):

    @cancellable
    async def get_device_list_changes_in_rooms(
-        self, room_ids: Collection[str], from_id: int
+        self, room_ids: Collection[str], from_id: int, to_id: int
    ) -> Optional[Set[str]]:
        """Return the set of users whose devices have changed in the given rooms
        since the given stream ID.
@@ -1475,7 +1462,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):

        sql = """
            SELECT DISTINCT user_id FROM device_lists_changes_in_room
-            WHERE {clause} AND stream_id >= ?
+            WHERE {clause} AND stream_id > ? AND stream_id <= ?
        """

        def _get_device_list_changes_in_rooms_txn(
@@ -1492,6 +1479,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
                self.database_engine, "room_id", chunk
            )
            args.append(from_id)
+            args.append(to_id)

            changes |= await self.db_pool.runInteraction(
                "get_device_list_changes_in_rooms",
--- a/synapse/storage/databases/main/room.py
+++ b/synapse/storage/databases/main/room.py
@@ -21,13 +21,11 @@
 #

 import logging
-from abc import abstractmethod
 from enum import Enum
 from typing import (
    TYPE_CHECKING,
    AbstractSet,
    Any,
-    Awaitable,
    Collection,
    Dict,
    List,
@@ -53,7 +51,7 @@ from synapse.api.room_versions import RoomVersion, RoomVersions
 from synapse.config.homeserver import HomeServerConfig
 from synapse.events import EventBase
 from synapse.replication.tcp.streams.partial_state import UnPartialStatedRoomStream
-from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
+from synapse.storage._base import db_to_json, make_in_list_sql_clause
 from synapse.storage.database import (
    DatabasePool,
    LoggingDatabaseConnection,
@@ -1684,6 +1682,58 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):

        return True

+    async def set_room_is_public(self, room_id: str, is_public: bool) -> None:
+        await self.db_pool.simple_update_one(
+            table="rooms",
+            keyvalues={"room_id": room_id},
+            updatevalues={"is_public": is_public},
+            desc="set_room_is_public",
+        )
+
+    async def set_room_is_public_appservice(
+        self, room_id: str, appservice_id: str, network_id: str, is_public: bool
+    ) -> None:
+        """Edit the appservice/network specific public room list.
+
+        Each appservice can have a number of published room lists associated
+        with them, keyed off of an appservice defined `network_id`, which
+        basically represents a single instance of a bridge to a third party
+        network.
+
+        Args:
+            room_id
+            appservice_id
+            network_id
+            is_public: Whether to publish or unpublish the room from the list.
+        """
+
+        if is_public:
+            await self.db_pool.simple_upsert(
+                table="appservice_room_list",
+                keyvalues={
+                    "appservice_id": appservice_id,
+                    "network_id": network_id,
+                    "room_id": room_id,
+                },
+                values={},
+                insertion_values={
+                    "appservice_id": appservice_id,
+                    "network_id": network_id,
+                    "room_id": room_id,
+                },
+                desc="set_room_is_public_appservice_true",
+            )
+        else:
+            await self.db_pool.simple_delete(
+                table="appservice_room_list",
+                keyvalues={
+                    "appservice_id": appservice_id,
+                    "network_id": network_id,
+                    "room_id": room_id,
+                },
+                desc="set_room_is_public_appservice_false",
+            )
+

 class _BackgroundUpdates:
    REMOVE_TOMESTONED_ROOMS_BG_UPDATE = "remove_tombstoned_rooms_from_directory"
@@ -1702,7 +1752,7 @@ _REPLACE_ROOM_DEPTH_SQL_COMMANDS = (
 )


-class RoomBackgroundUpdateStore(SQLBaseStore):
+class RoomBackgroundUpdateStore(RoomWorkerStore):
    def __init__(
        self,
        database: DatabasePool,
@@ -1935,14 +1985,6 @@ class RoomBackgroundUpdateStore(SQLBaseStore):

        return len(rooms)

-    @abstractmethod
-    def set_room_is_public(self, room_id: str, is_public: bool) -> Awaitable[None]:
-        # this will need to be implemented if a background update is performed with
-        # existing (tombstoned, public) rooms in the database.
-        #
-        # It's overridden by RoomStore for the synapse master.
-        raise NotImplementedError()
-
    async def has_auth_chain_index(self, room_id: str) -> bool:
        """Check if the room has (or can have) a chain cover index.

@@ -2349,62 +2391,6 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore):
            },
        )

-    async def set_room_is_public(self, room_id: str, is_public: bool) -> None:
-        await self.db_pool.simple_update_one(
-            table="rooms",
-            keyvalues={"room_id": room_id},
-            updatevalues={"is_public": is_public},
-            desc="set_room_is_public",
-        )
-
-        self.hs.get_notifier().on_new_replication_data()
-
-    async def set_room_is_public_appservice(
-        self, room_id: str, appservice_id: str, network_id: str, is_public: bool
-    ) -> None:
-        """Edit the appservice/network specific public room list.
-
-        Each appservice can have a number of published room lists associated
-        with them, keyed off of an appservice defined `network_id`, which
-        basically represents a single instance of a bridge to a third party
-        network.
-
-        Args:
-            room_id
-            appservice_id
-            network_id
-            is_public: Whether to publish or unpublish the room from the list.
-        """
-
-        if is_public:
-            await self.db_pool.simple_upsert(
-                table="appservice_room_list",
-                keyvalues={
-                    "appservice_id": appservice_id,
-                    "network_id": network_id,
-                    "room_id": room_id,
-                },
-                values={},
-                insertion_values={
-                    "appservice_id": appservice_id,
-                    "network_id": network_id,
-                    "room_id": room_id,
-                },
-                desc="set_room_is_public_appservice_true",
-            )
-        else:
-            await self.db_pool.simple_delete(
-                table="appservice_room_list",
-                keyvalues={
-                    "appservice_id": appservice_id,
-                    "network_id": network_id,
-                    "room_id": room_id,
-                },
-                desc="set_room_is_public_appservice_false",
-            )
-
-        self.hs.get_notifier().on_new_replication_data()
-
    async def add_event_report(
        self,
        room_id: str,
--- a/tests/rest/synapse/init.py
+++ b/tests/rest/synapse/init.py
@@ -0,0 +1,12 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2024 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
--- a/tests/rest/synapse/client/init.py
+++ b/tests/rest/synapse/client/init.py
@@ -0,0 +1,12 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2024 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
--- a/tests/rest/synapse/client/test_federation_whitelist.py
+++ b/tests/rest/synapse/client/test_federation_whitelist.py
@@ -0,0 +1,119 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2024 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
+
+from typing import Dict
+
+from twisted.web.resource import Resource
+
+from synapse.rest import admin
+from synapse.rest.client import login
+from synapse.rest.synapse.client import build_synapse_client_resource_tree
+
+from tests import unittest
+
+
+class FederationWhitelistTests(unittest.HomeserverTestCase):
+    servlets = [
+        admin.register_servlets_for_client_rest_resource,
+        login.register_servlets,
+    ]
+
+    def create_resource_dict(self) -> Dict[str, Resource]:
+        base = super().create_resource_dict()
+        base.update(build_synapse_client_resource_tree(self.hs))
+        return base
+
+    def test_default(self) -> None:
+        "If the config option is not enabled, the endpoint should 404"
+        channel = self.make_request(
+            "GET", "/_synapse/client/v1/config/federation_whitelist", shorthand=False
+        )
+
+        self.assertEqual(channel.code, 404)
+
+    @unittest.override_config({"federation_whitelist_endpoint_enabled": True})
+    def test_no_auth(self) -> None:
+        "Endpoint requires auth when enabled"
+
+        channel = self.make_request(
+            "GET", "/_synapse/client/v1/config/federation_whitelist", shorthand=False
+        )
+
+        self.assertEqual(channel.code, 401)
+
+    @unittest.override_config({"federation_whitelist_endpoint_enabled": True})
+    def test_no_whitelist(self) -> None:
+        "Test when there is no whitelist configured"
+
+        self.register_user("user", "password")
+        tok = self.login("user", "password")
+
+        channel = self.make_request(
+            "GET",
+            "/_synapse/client/v1/config/federation_whitelist",
+            shorthand=False,
+            access_token=tok,
+        )
+
+        self.assertEqual(channel.code, 200)
+        self.assertEqual(
+            channel.json_body, {"whitelist_enabled": False, "whitelist": []}
+        )
+
+    @unittest.override_config(
+        {
+            "federation_whitelist_endpoint_enabled": True,
+            "federation_domain_whitelist": ["example.com"],
+        }
+    )
+    def test_whitelist(self) -> None:
+        "Test when there is a whitelist configured"
+
+        self.register_user("user", "password")
+        tok = self.login("user", "password")
+
+        channel = self.make_request(
+            "GET",
+            "/_synapse/client/v1/config/federation_whitelist",
+            shorthand=False,
+            access_token=tok,
+        )
+
+        self.assertEqual(channel.code, 200)
+        self.assertEqual(
+            channel.json_body, {"whitelist_enabled": True, "whitelist": ["example.com"]}
+        )
+
+    @unittest.override_config(
+        {
+            "federation_whitelist_endpoint_enabled": True,
+            "federation_domain_whitelist": ["example.com", "example.com"],
+        }
+    )
+    def test_whitelist_no_duplicates(self) -> None:
+        "Test when there is a whitelist configured with duplicates, no duplicates are returned"
+
+        self.register_user("user", "password")
+        tok = self.login("user", "password")
+
+        channel = self.make_request(
+            "GET",
+            "/_synapse/client/v1/config/federation_whitelist",
+            shorthand=False,
+            access_token=tok,
+        )
+
+        self.assertEqual(channel.code, 200)
+        self.assertEqual(
+            channel.json_body, {"whitelist_enabled": True, "whitelist": ["example.com"]}
+        )
Author	SHA1	Message	Date
Erik Johnston	ad3da48636	Newsfile	2024-05-15 15:14:34 +01:00
Erik Johnston	5090acc46f	Newsfile	2024-05-15 15:14:21 +01:00
Erik Johnston	2b8e932749	Cap the top stream ID when fetching changed devices	2024-05-15 15:11:50 +01:00
Andrew Morgan	68dca8076f	Merge branch 'master' into develop	2024-05-14 15:21:42 +01:00
Erik Johnston	284d85dee3	Cache literal sync filter validation (#17186 ) The sliding sync proxy (amongst other things) use literal json blobs as filters, and repeatedly validating them takes a bunch of CPU.	2024-05-14 15:08:46 +01:00
Erik Johnston	ebe77381b0	Reduce pauses on large device list changes (#17192 ) For large accounts waking up all the relevant notifier streams can cause pauses of the reactor.	2024-05-14 14:39:11 +01:00
Erik Johnston	0b91ccce47	Improve perf of sync device lists (#17191 ) It's almost always more efficient to query the rooms that have device list changes, rather than looking at the list of all users whose devices have changed and then look for shared rooms.	2024-05-14 14:39:04 +01:00
Andrew Morgan	ecf4e0674c	1.107.0	2024-05-14 14:15:41 +01:00
Aurélien Grimpard	7d82987b27	Allows CAS SSO flow to provide user IDs composed of numbers only (#17098 )	2024-05-14 13:55:32 +01:00
dependabot[bot]	bd8d8865fb	Bump serde_json from 1.0.116 to 1.0.117 (#17182 )	2024-05-14 09:32:23 +01:00
dependabot[bot]	caf528477e	Bump serde from 1.0.200 to 1.0.201 (#17183 )	2024-05-14 09:32:14 +01:00
dependabot[bot]	f0c72d8e87	Bump gitpython from 3.1.41 to 3.1.43 (#17181 )	2024-05-14 09:32:01 +01:00
dependabot[bot]	03a342b049	Bump cryptography from 42.0.5 to 42.0.7 (#17180 )	2024-05-14 09:31:46 +01:00
dependabot[bot]	aa6345cb3b	Bump immutabledict from 4.1.0 to 4.2.0 (#17179 )	2024-05-14 09:31:35 +01:00
dependabot[bot]	2b438df9b3	Bump sentry-sdk from 1.40.3 to 2.1.1 (#17178 )	2024-05-14 09:31:25 +01:00
Erik Johnston	038b9ec59a	An federation whitelist query endpoint extension (#16848 ) This is to allow clients to query the configured federation whitelist. Disabled by default. --------- Co-authored-by: Devon Hudson <devonhudson@librem.one> Co-authored-by: devonh <devon.dmytro@gmail.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-05-13 19:38:45 +00:00
Erik Johnston	59ac541310	Actually fix public rooms (#17184 ) See #17177. I'm an idiot and moved them to the wrong store 🤦	2024-05-13 13:11:07 +01:00
Erik Johnston	a2e6f43f11	Fix bug with creating public rooms on workers (#17177 ) If room publication is disabled then creating public rooms on workers would not work. Introduced in #16811.	2024-05-13 12:12:26 +01:00
ll-SKY-ll	4cf4a8281b	Update docs to bump libjemalloc version on latest debian; correct "push_rules" stream name (#17171 )	2024-05-09 10:50:05 +00:00
Andrew Morgan	ef7e040e54	Bump black from 24.2.0 to 24.4.2 (#17170 )	2024-05-08 17:02:09 +01:00
devonh	393429d692	Fix undiscovered linter errors (#17166 ) Linter errors are showing up in #17147 that are unrelated to that PR. The errors do not currently show up on develop. This PR aims to resolve the linter errors separately from #17147.	2024-05-08 14:57:32 +00:00
				`@@ -0,0 +1 @@`
				`Add a feature that allows clients to query the configured federation whitelist. Disabled by default.`
				`@@ -0,0 +1 @@`
				`Add the ability to allow numeric user IDs with a specific prefix when in the CAS flow. Contributed by Aurélien Grimpard.`
				`@@ -0,0 +1 @@`
				`Update the Admin FAQ with the current libjemalloc version for latest Debian stable. Additionally update the name of the "push_rules" stream in the Workers documentation.`
				`@@ -0,0 +1 @@`
				`Fix bug where disabling room publication prevented public rooms being created on workers.`
				`@@ -0,0 +1 @@`
				`Cache literal sync filter validation for performance.`
				`@@ -0,0 +1 @@`
				Improve performance of calculating device lists changes in `/sync`.
				`@@ -0,0 +1 @@`
				`Improve performance by fixing a reactor pause.`
				`@@ -0,0 +1 @@`
				Fix bug where we sent down redundant device list changes down `/sync`.