We do this by a) not pulling out all membership events, and b) batch
inserting bans.
One blocking concern is that this bypasses the `update_membership`
function, which otherwise all other membership events go via. In this
case it's fine (having audited what it is doing), but I'm hesitant to
set the precedent of bypassing it, given it has a lot of logic in there.
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Best reviewed commit by commit.
With the new dedicated MAS API
(https://github.com/element-hq/synapse/pull/18520), it's possible that
deactivation starts off the main process, which was not possible because
of a few calls.
I basically looked at everything that the deactivation handler was
doing, reviewed whether it could run on workers or not, and find a
workaround when possible
---------
Co-authored-by: Eric Eastwood <erice@element.io>
The case where a consumer stops downloading media that is currently
being streamed is now able to be handled explicitly.
That scenario isn't really an error, it is expected behaviour.
This PR adds a custom exception which allows us to drop the log level
for this specific case from `WARNING` to `INFO`.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Eric Eastwood <erice@element.io>
This introduces a dedicated API for MAS to consume. Companion PR on the
MAS side: element-hq/matrix-authentication-service#4801
This has a few advantages over the previous admin API:
- it works on workers (this will be documented once we stabilise MSC3861
as a whole)
- it is more efficient because more focused
- it propagates trace contexts from MAS
- it is only accessible to MAS (through the shared secret) and will let
us remove the weird hack that made this token 'admin' with a ghost
'@__oidc_admin:' user
The next MAS version should support it, but will be opt-in. The version
after that should use this new API by default
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Spawning from https://github.com/element-hq/synapse/pull/18689
Example CI failure that will stop people from leaving stray `Cargo.lock`
changes behind,
```
Error: Cargo.lock has uncommitted changes after install. Please run 'poetry install --extras all' and commit the Cargo.lock changes.
```
The main goal of this PR is to handle device list changes onto multiple
writers, off the main process, so that we can have logins happening
whilst Synapse is rolling-restarting.
This is quite an intrusive change, so I would advise to review this
commit by commit; I tried to keep the history as clean as possible.
There are a few things to consider:
- the `device_list_key` in stream tokens becomes a
`MultiWriterStreamToken`, which has a few implications in sync and on
the storage layer
- we had a split between `DeviceHandler` and `DeviceWorkerHandler` for
master vs. worker process. I've kept this split, but making it rather
writer vs. non-writer worker, using method overrides for doing
replication calls when needed
- there are a few operations that need to happen on a single worker at a
time. Instead of using cross-worker locks, for now I made them run on
the first writer on the list
---------
Co-authored-by: Eric Eastwood <erice@element.io>
Normal install results in `Cargo.lock` changes constantly popping up for me as I navigate my
branches. This was probably caused by some Depdendabot PR updating the
`Cargo.toml` without `Cargo.lock` or something.
```
poetry install --extras all
```
In another PR, I've also added CI to ensure we don't leave `Cargo.lock`
changes behind to avoid this annoyance in the future ->
https://github.com/element-hq/synapse/pull/18693
Clean up `MetricsResource`, Prometheus hacks
(`_set_prometheus_client_use_created_metrics`), and better document why
we care about having a separate `metrics` listener type.
These clean-up changes have been split out from
https://github.com/element-hq/synapse/pull/18584 since that PR was
closed.
Bumps [lxml](https://github.com/lxml/lxml) from 5.4.0 to 6.0.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/lxml/lxml/blob/master/CHANGES.txt">lxml's
changelog</a>.</em></p>
<blockquote>
<h1>6.0.0 (2025-06-26)</h1>
<h2>Features added</h2>
<ul>
<li>
<p>GH#463: <code>lxml.html.diff</code> is faster and provides
structurally better diffs.
Original patch by Steven Fernandez.</p>
</li>
<li>
<p>GH#405: The factories <code>Element</code> and
<code>ElementTree</code> can now be used in type hints.</p>
</li>
<li>
<p>GH#448: Parsing from <code>memoryview</code> and other buffers is
supported to allow zero-copy parsing.</p>
</li>
<li>
<p>GH#437: <code>lxml.html.builder</code> was missing several HTML5 tag
names.
Patch by Nick Tarleton.</p>
</li>
<li>
<p>GH#458: <code>CDATA</code> can now be written into the incremental
<code>xmlfile()</code> writer.
Original patch by Lane Shaw.</p>
</li>
<li>
<p>A new parser option <code>decompress=False</code> was added that
controls the automatic
input decompression when using libxml2 2.15.0 or later. Disabling this
option
by default will effectively prevent decompression bombs when handling
untrusted
input. Code that depends on automatic decompression must enable this
option.
Note that libxml2 2.15.0 was not released yet, so this option currently
has no
effect but can already be used.</p>
</li>
<li>
<p>The set of compile time / runtime supported libxml2 feature names is
available as
<code>etree.LIBXML_COMPILED_FEATURES</code> and
<code>etree.LIBXML_FEATURES</code>.
This currently includes
<code>catalog</code>, <code>ftp</code>, <code>html</code>,
<code>http</code>, <code>iconv</code>, <code>icu</code>,
<code>lzma</code>, <code>regexp</code>, <code>schematron</code>,
<code>xmlschema</code>, <code>xpath</code>, <code>zlib</code>.</p>
</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li>
<p>GH#353: Predicates in <code>.find*()</code> could mishandle tag
indices if a default namespace is provided.
Original patch by Luise K.</p>
</li>
<li>
<p>GH#272: The <code>head</code> and <code>body</code> properties of
<code>lxml.html</code> elements failed if no such element
was found. They now return <code>None</code> instead.
Original patch by FVolral.</p>
</li>
<li>
<p>Tag names provided by code (API, not data) that are longer than
<code>INT_MAX</code>
could be truncated or mishandled in other ways.</p>
</li>
<li>
<p><code>.text_content()</code> on <code>lxml.html</code> elements
accidentally returned a "smart string"
without additional information. It now returns a plain string.</p>
</li>
<li>
<p>LP#2109931: When building lxml with coverage reporting, it now
disables the <code>sys.monitoring</code>
support due to the lack of support in <a
href="https://redirect.github.com/nedbat/coveragepy/issues/1790">nedbat/coveragepy#1790</a></p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="2a67034bc2"><code>2a67034</code></a>
Prepare release of 6.0.0.</li>
<li><a
href="e0b4e02182"><code>e0b4e02</code></a>
Update changelog.</li>
<li><a
href="d3f4dcf689"><code>d3f4dcf</code></a>
Build: Upgrade libxml2 to latest 2.14.4.</li>
<li><a
href="014e51cce3"><code>014e51c</code></a>
Build: Add Windows arm64 wheel builds (<a
href="https://redirect.github.com/lxml/lxml/issues/465">GH-465</a>)</li>
<li><a
href="d3914dcb6a"><code>d3914dc</code></a>
Only use "xmlCtxtIsStopped()" from libxml2 2.15.0 on since it
fails to cover ...</li>
<li><a
href="6e41390275"><code>6e41390</code></a>
Avoid reading the deprecated "disableSAX" attribute of
"xmlParserCtxt".</li>
<li><a
href="f85da81b1d"><code>f85da81</code></a>
Use newer "language_level=3" in ElementPath module.</li>
<li><a
href="787315eb54"><code>787315e</code></a>
Build: bump pypa/cibuildwheel in the github-actions group (<a
href="https://redirect.github.com/lxml/lxml/issues/464">#464</a>)</li>
<li><a
href="fb3adb1dce"><code>fb3adb1</code></a>
Readme: Add project income report for 2024.</li>
<li><a
href="8e61a757c8"><code>8e61a75</code></a>
Fit cached tuple more nicely into cachelines.</li>
<li>Additional commits viewable in <a
href="https://github.com/lxml/lxml/compare/lxml-5.4.0...lxml-6.0.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Fixes https://github.com/element-hq/synapse/issues/18659
This changes the Tokio runtime to be attached to the Twisted reactor.
This way, the Tokio runtime starts when the Twisted reactor starts, and
*not* when the module gets loaded.
This is important as starting the runtime on module load meant that it
broke when Synapse was started with `daemonize`/`synctl`, as forks only
retain the calling threads, breaking the Tokio runtime.
This also changes so that the HttpClient gets the Twisted reactor
explicitly as parameter instead of loading it from
`twisted.internet.reactor`
