1
0

Compare commits

...

98 Commits

Author SHA1 Message Date
Eric Eastwood
b732dabe3d Try poetry lock --no-update again after clear caches
See https://github.com/matrix-org/synapse/pull/13399#discussion_r930442775
2022-07-26 16:55:57 -05:00
Eric Eastwood
75a751dd54 poetry.lock from develop 2022-07-26 16:32:13 -05:00
Eric Eastwood
c15e0bd54a Try poetry lock --no-update 2022-07-26 16:25:51 -05:00
Eric Eastwood
801069ae81 Draft: Try use rust-jaeger-python-client 2022-07-26 16:20:20 -05:00
Eric Eastwood
522c29bfc7 Instrument /messages for understandable traces 2022-07-22 22:29:18 -05:00
Eric Eastwood
357561c1a2 Backfill remote event fetched by MSC3030 so we can paginate from it later (#13205)
Depends on https://github.com/matrix-org/synapse/pull/13320

Complement tests: https://github.com/matrix-org/complement/pull/406

We could use the same method to backfill for `/context` as well in the future, see https://github.com/matrix-org/synapse/issues/3848
2022-07-22 16:00:11 -05:00
Richard van der Hoff
c7c84b81e3 Update config_documentation.md (#13364)
"changed in" goes before the example
2022-07-22 13:50:20 +01:00
Sean Quah
0fa41a7b17 Update locked frozendict version to 2.3.3 (#13352)
frozendict 2.3.3 includes fixes for memory leaks that get triggered during `/sync`.
2022-07-22 10:26:09 +01:00
Sean Quah
158782c3ce Skip soft fail checks for rooms with partial state (#13354)
When a room has the partial state flag, we may not have an accurate
`m.room.member` event for event senders in the room's current state, and
so cannot perform soft fail checks correctly. Skip the soft fail check
entirely in this case.

As an alternative, we could block until we have full state, but that
would prevent us from receiving incoming events over federation, which
is undesirable.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-22 10:13:01 +01:00
Nick Mills-Barrett
86e366a46e Remove old empty/redundant slaved stores. (#13349) 2022-07-21 17:56:45 +00:00
Erik Johnston
0b87eb8e0c Make DictionaryCache have better expiry properties (#13292) 2022-07-21 17:13:44 +01:00
Erik Johnston
13341dde5a Don't hold onto full state in state cache (#13324) 2022-07-21 16:02:02 +01:00
Brendan Abolivier
10e4093839 Call out buildkit is required when building test docker images (#13338)
Co-authored-by: David Robertson <davidr@element.io>
2022-07-21 14:29:58 +02:00
David Robertson
34949ead1f Track DB txn times w/ two counters, not histogram (#13342) 2022-07-21 13:23:05 +01:00
Patrick Cloke
50122754c8 Add missing types to opentracing. (#13345)
After this change `synapse.logging` is fully typed.
2022-07-21 12:01:52 +00:00
Nick Mills-Barrett
190f49d8ab Use cache store remove base slaved (#13329)
This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
2022-07-21 11:51:30 +01:00
David Robertson
4f57ef0b18 Merge branch 'master' into develop 2022-07-21 11:27:08 +01:00
David Teller
b909d5327b Document rc_invites.per_issuer, added in v1.63.
Resolves #13330.
Missed in #13125.

Signed-off-by: David Teller <davidt@element.io>
2022-07-21 11:26:34 +01:00
Eric Eastwood
0f971ca68e Update get_pdu to return the original, pristine EventBase (#13320)
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier`  and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call.

Split out from https://github.com/matrix-org/synapse/pull/13205

As discussed at:

 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746
 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125

Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](7864f33e28/synapse/federation/federation_client.py (L581-L594)).
2022-07-20 15:58:51 -05:00
Shay
a1b62af2af Validate federation destinations and log an error if server name is invalid. (#13318) 2022-07-20 11:17:26 -07:00
Erik Johnston
d3995049a8 Merge remote-tracking branch 'origin/master' into develop 2022-07-20 14:59:43 +01:00
Erik Johnston
93740cae57 1.63.1 2022-07-20 13:37:00 +01:00
Erik Johnston
b4ae3b0d44 Don't include appservice users when calculating push rules (#13332)
This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078
2022-07-20 12:06:13 +01:00
Sean Quah
172ce29b14 Fix spurious warning when fetching state after a missing prev event (#13258) 2022-07-19 19:15:54 +01:00
Patrick Cloke
a6895dd576 Add type annotations to trace decorator. (#13328)
Functions that are decorated with `trace` are now properly typed
and the type hints for them are fixed.
2022-07-19 14:14:30 -04:00
Brendan Abolivier
47822fd2e8 Merge branch 'master' into develop 2022-07-19 16:14:02 +02:00
Erik Johnston
de70b25e84 Reduce memory usage of state group cache (#13323) 2022-07-19 14:40:37 +01:00
Patrick Cloke
1efe6b8c41 Stop building Ubuntu 21.10 (Impish Indri) which is end of life. (#13326) 2022-07-19 09:08:46 -04:00
Brendan Abolivier
6fccd72f42 Improve precision on validation improvements 2022-07-19 14:53:12 +02:00
Brendan Abolivier
097afd0e0b 1.63.0 2022-07-19 14:43:28 +02:00
Andrew Morgan
6faaf76a32 Remove 'anonymised' from the phone home stats documentation (#13321) 2022-07-19 12:38:29 +00:00
villepeh
84c5e6b1fd Bash script for creating multiple stream writers (#13271)
Add another bash script to the contrib directory. It creates multiple stream writers and also prints out the example configuration for homeserver.yaml.

Signed-off-by: Ville Petteri Huh.
2022-07-19 12:37:20 +00:00
Jörg Behrmann
87a917e8c8 Add notes when config options were changed to config documentation (#13314)
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-07-19 12:36:29 +00:00
David Robertson
b977867358 Rate limit joins per-room (#13276) 2022-07-19 11:45:17 +00:00
Nick Mills-Barrett
2ee0b6ef4b Safe async event cache (#13308)
Fix race conditions in the async cache invalidation logic, by separating
the async & local invalidation calls and ensuring any async call i
executed first.

Signed off by Nick @ Beeper (@Fizzadar).
2022-07-19 11:25:29 +00:00
Shay
7864f33e28 Increase batch size of bulk_get_push_rules and _get_joined_profiles_from_event_ids. (#13300) 2022-07-18 13:15:23 -07:00
Shay
15edf23626 Improve performance of query _get_subset_users_in_room_with_profiles (#13299) 2022-07-18 12:35:45 -07:00
Sean Quah
5526f9fc4f Fix overcounting of pushers when they are replaced (#13296)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-18 17:39:39 +01:00
Brendan Abolivier
8c60c572f0 Up the dependency on canonicaljson to ^1.5.0 (#13172)
Co-authored-by: David Robertson <davidr@element.io>
2022-07-18 17:30:59 +02:00
Andrew Morgan
bb25dd81e3 Prevent #3679 from appearing in blame results (#13311) 2022-07-18 14:02:32 +00:00
Erik Johnston
f721f1baba Revert "Make all process_replication_rows methods async (#13304)" (#13312)
This reverts commit 5d4028f217.
2022-07-18 14:28:14 +01:00
Erik Johnston
cf5fa5063d Don't pull out full state when sending dummy events (#13310) 2022-07-18 14:19:11 +01:00
Nick Mills-Barrett
6785b0f39d Use READ COMMITTED isolation level when purging rooms (#12942)
To close: #10294.

Signed off by Nick @ Beeper.
2022-07-18 14:17:24 +01:00
Andrew Morgan
c5f487b7cb Update expected DB query count when creating a room (#13307) 2022-07-18 13:02:25 +01:00
Erik Johnston
c6a05063ff Don't pull out the full state when creating an event (#13281) 2022-07-18 10:05:30 +01:00
Dirk Klimpel
efee345b45 Remove unnecessary json.dumps from tests (#13303) 2022-07-17 22:28:45 +01:00
Nick Mills-Barrett
5d4028f217 Make all process_replication_rows methods async (#13304)
More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so).

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-17 22:19:43 +01:00
Dirk Klimpel
96cf81e312 Use HTTPStatus constants in place of literals in tests. (#13297) 2022-07-15 19:31:27 +00:00
Eric Eastwood
7b67e93d49 Provide more info why we don't have any thumbnails to serve (#13038)
Fix https://github.com/matrix-org/synapse/issues/13016

## New error code and status

### Before

Previously, we returned a `404` for `/thumbnail` which isn't even in the spec.

```json
{
  "errcode": "M_NOT_FOUND",
  "error": "Not found [b'hs1', b'tefQeZhmVxoiBfuFQUKRzJxc']"
}
```

### After

What does the spec say?

> 400: The request does not make sense to the server, or the server cannot thumbnail the content. For example, the client requested non-integer dimensions or asked for negatively-sized images.
>
> *-- https://spec.matrix.org/v1.1/client-server-api/#get_matrixmediav3thumbnailservernamemediaid*

Now with this PR, we respond with a `400` when we don't have thumbnails to serve and we explain why we might not have any thumbnails.

```json
{
    "errcode": "M_UNKNOWN",
    "error": "Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)",
}
```

> Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)


---

We still respond with a 404 in many other places. But we can iterate on those later and maybe keep some in some specific places after spec updates/clarification: https://github.com/matrix-org/matrix-spec/issues/1122

We can also iterate on the bugs where Synapse doesn't thumbnail when it should in other issues/PRs.
2022-07-15 11:42:21 -05:00
David Robertson
e9ce4d089b Use and recommend poetry 1.1.14, up from 1.1.12 (#13285) 2022-07-15 16:18:47 +01:00
Erik Johnston
0731e0829c Don't pull out the full state when storing state (#13274) 2022-07-15 12:59:45 +00:00
Patrick Cloke
3343035a06 Use a real room in the notification rotation tests. (#13260)
Instead of manually inserting fake data. This fixes some issues with
having to manually calculate stream orderings and other oddities.
2022-07-15 08:22:43 -04:00
David Robertson
7281591f4c Use state before join to determine if we _should_perform_remote_join (#13270)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-07-15 12:20:47 +00:00
Sean Quah
d765ada84f Update locked frozendict version to 2.3.2 (#13284)
`frozendict` 2.3.2 includes a fix for a memory leak in
`frozendict.__hash__`. This likely has no impact outside of the
deprecated `/initialSync` endpoint, which uses `StreamToken`s,
containing `RoomStreamToken`s, containing `frozendict`s, as cache keys.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-15 13:18:51 +01:00
Richard van der Hoff
b116d3ce00 Bg update to populate new events table columns (#13215)
These columns were added back in Synapse 1.52, and have been populated for new
events since then. It's now (beyond) time to back-populate them for existing
events.
2022-07-15 12:47:26 +01:00
Erik Johnston
7be954f59b Fix a bug which could lead to incorrect state (#13278)
There are two fixes here:
1. A long-standing bug where we incorrectly calculated `delta_ids`; and
2. A bug introduced in #13267 where we got current state incorrect.
2022-07-15 11:06:41 +00:00
Richard van der Hoff
512486bbeb Docker: copy postgres from base image (#13279)
When building the docker images for complement testing, copy a preinstalled
complement over from a base image, rather than apt installing it. This avoids
network traffic and is much faster.
2022-07-15 11:13:40 +01:00
Nick Mills-Barrett
cc21a431f3 Async get event cache prep (#13242)
Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches.

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-15 09:30:46 +00:00
Nick Mills-Barrett
21eeacc995 Federation Sender & Appservice Pusher Stream Optimisations (#13251)
* Replace `get_new_events_for_appservice` with `get_all_new_events_stream`

The functions were near identical and this brings the AS worker closer
to the way federation senders work which can allow for multiple workers
to handle AS traffic.

* Pull received TS alongside events when processing the stream

This avoids an extra query -per event- when both federation sender
and appservice pusher process events.
2022-07-15 09:36:56 +01:00
Richard van der Hoff
fe15a865a5 Rip out auth-event reconciliation code (#12943)
There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution.

This isn't specced, and there's general agreement we shouldn't be doing it.

However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
2022-07-14 21:52:26 +00:00
Richard van der Hoff
df55b377be CHANGES.md: fix link to upgrade notes 2022-07-14 15:07:52 +01:00
Erik Johnston
0ca4172b5d Don't pull out state in compute_event_context for unconflicted state (#13267) 2022-07-14 13:57:02 +00:00
David Robertson
599c403d99 Allow rate limiters to passively record actions they cannot limit (#13253)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-07-13 19:09:42 +00:00
David Robertson
0eb7e69768 Notifier: accept callbacks to fire on room joins (#13254) 2022-07-13 19:48:24 +01:00
Jacek Kuśnierz
cc1071598a Call the v2 identity service /3pid/unbind endpoint, rather than v1. (#13240)
* Drop support for v1 unbind

Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de>

* Add changelog

Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de>

* Update changelog.d/13240.misc
2022-07-13 19:43:17 +01:00
Shay
ad5761b65c Add support for room version 10 (#13220) 2022-07-13 11:36:02 -07:00
jejo86
2341032cf2 Document advising against publicly exposing the Admin API and provide a usage example (#13231)
* Admin API request explanation improved

Pointed out, that the Admin API is not accessible by default from any remote computer, but only from the PC `matrix-synapse` is running on.
Added a full, working example, making sure to include the cURL flag `-X`, which needs to be prepended to `GET`, `POST`, `PUT` etc. and listing the full query string including protocol, IP address and port.

* Admin API request explanation improved

* Apply suggestions from code review

Update changelog. Reword prose.

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-07-13 19:33:33 +01:00
Nick Mills-Barrett
982fe29655 Optimise room creation event lookups part 2 (#13224) 2022-07-13 19:32:46 +01:00
Patrick Cloke
1d5c80b161 Reduce duplicate code in receipts servlets. (#13198) 2022-07-13 13:23:16 -04:00
Brad Murray
3371e1abcb Add prometheus counters for content types other than events (#13175) 2022-07-13 15:18:20 +01:00
Patrick Cloke
4db7862e0f Drop unused tables from groups/communities. (#12967)
These tables have been unused since Synapse v1.61.0, although schema version 72
was added in Synapse v1.62.0.
2022-07-13 09:55:14 -04:00
Patrick Cloke
90e9b4fa1e Do not fail build if complement with workers fails. (#13266) 2022-07-13 08:30:42 -04:00
Thomas Weston
0312ff44c6 Fix "add user" admin api error when request contains a "msisdn" threepid (#13263)
Co-authored-by: Thomas Weston <thomas.weston@clearspancloud.com>
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-07-13 11:33:21 +01:00
Patrick Cloke
1381563988 Inline URL preview documentation. (#13261)
Inline URL preview documentation near the implementation.
2022-07-12 15:01:58 -04:00
Richard van der Hoff
a366b75b72 Drop unused table event_reference_hashes (#13218)
This is unused since Synapse 1.60.0 (#12679). It's time for it to go.
2022-07-12 18:52:06 +00:00
Jacek Kuśnierz
7218a0ca18 Drop support for calling /_matrix/client/v3/account/3pid/bind without an id_access_token (#13239)
Fixes #13201

Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-07-12 18:48:29 +00:00
David Robertson
52a0c8f2f7 Rename test case method to add_hashes_and_signatures_from_other_server (#13255) 2022-07-12 18:46:32 +00:00
Richard van der Hoff
fa71bb18b5 Drop support for delegating email validation (#13192)
* Drop support for delegating email validation

Delegating email validation to an IS is insecure (since it allows the owner of
the IS to do a password reset on your HS), and has long been deprecated. It
will now cause a config error at startup.

* Update unit test which checks for email verification

Give it an `email` config instead of a threepid delegate

* Remove unused method `requestEmailToken`

* Simplify config handling for email verification

Rather than an enum and a boolean, all we need here is a single bool, which
says whether we are or are not doing email verification.

* update docs

* changelog

* upgrade.md: fix typo

* update version number

this will be in 1.64, not 1.63

* update version number

this one too
2022-07-12 19:18:53 +01:00
Sean Quah
3f178332d6 Log the stack when waiting for an entire room to be un-partial stated (#13257)
The stack is already logged when waiting for an event to be un-partial
stated. Log the stack for rooms as well, to aid in debugging.
2022-07-12 18:57:38 +01:00
Shay
6f30eb5b8e Add info about configuration in the url preview docs (#13233)
Cross-link doc pages for easier navigation.
2022-07-12 13:48:47 -04:00
Quentin Gliech
b19060a29b Make the AS login method call Auth.get_user_by_req for checking the AS token. (#13094)
This gets rid of another usage of get_appservice_by_req, with all the benefits, including correctly tracking the appservice IP and setting the tracing attributes correctly.

Signed-off-by: Quentin Gliech <quenting@element.io>
2022-07-12 18:06:29 +01:00
andrew do
2d82cdafd2 expose whether a room is a space in the Admin API (#13208) 2022-07-12 15:30:53 +01:00
Sean Quah
f14c632134 Update changelog once more 2022-07-12 13:01:42 +01:00
Sean Quah
ac7aec0cd3 Reorder and tidy up changelog 2022-07-12 12:52:47 +01:00
Sean Quah
6173d585df 1.63.0rc1 2022-07-12 11:26:25 +01:00
Erik Johnston
e5716b631c Don't pull out the full state when calculating push actions (#13078) 2022-07-11 20:08:39 +00:00
villepeh
bc8eefc1e1 Add a sample bash script to docs for creating multiple worker files (#13032)
Signed-off-by: Ville Petteri Huh.
2022-07-11 18:33:53 +01:00
Nick Mills-Barrett
92202ce867 Reduce event lookups during room creation by passing known event IDs (#13210)
Inspired by the room batch handler, this uses previous event inserts to
pre-populate prev events during room creation, reducing the number of
queries required to create a room.

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-11 18:00:12 +01:00
David Teller
11f811470f Uniformize spam-checker API, part 5: expand other spam-checker callbacks to return Tuple[Codes, dict] (#13044)
Signed-off-by: David Teller <davidt@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-07-11 16:52:10 +00:00
Travis Ralston
d736d5cfad Fix to-device messages not being sent to MSC3202-enabled appservices (#13235)
The field name was simply incorrect, leading to errors.
2022-07-11 17:22:17 +01:00
Erik Johnston
f1711e1f5c Remove delay when rotating event push actions (#13211)
We want to be as up to date as possible, and sleeping doesn't help here
and can mean we fall behind.
2022-07-11 16:51:30 +01:00
Andrew Morgan
5ef2f87569 Document the 'databases' homeserver config option (#13212) 2022-07-11 14:05:24 +00:00
Erik Johnston
e610128c50 Add a filter_event_for_clients_with_state function (#13222) 2022-07-11 14:14:09 +01:00
Travis Ralston
a113011794 Fix appservice EDUs failing to send if the EDU doesn't have a room ID (#13236)
* Fix appservice EDUs failing to send if the EDU doesn't have a room ID

As is in the case of presence.

* changelog

* linter

* fix linter again
2022-07-11 14:12:28 +01:00
David Robertson
28d96cb2b4 Ensure portdb selects _all_ rows with negative rowids (#13226) 2022-07-11 10:36:18 +01:00
Sumner Evans
739adf1551 editorconfig: add max_line_length for Python files (#13228)
See the documentation for the property here:
https://github.com/editorconfig/editorconfig/wiki/EditorConfig-Properties#max_line_length

Signed-off-by: Sumner Evans <me@sumnerevans.com>
2022-07-08 16:40:25 +00:00
Erik Johnston
757bc0caef Fix notification count after a highlighted message (#13223)
Fixes #13196

Broke by #13005
2022-07-08 14:00:29 +01:00
Eric Eastwood
a962c5a56d Fix exception when using MSC3030 to look for remote federated events before room creation (#13197)
Complement tests: https://github.com/matrix-org/complement/pull/405

This happens when you have some messages imported before the room is created.
Then use MSC3030 to look backwards before the room creation from a remote
federated server. The server won't find anything locally, but will ask over
federation which will have the remote event. The previous logic would
choke on not having the local event assigned.

```
Failed to fetch /timestamp_to_event from hs2 because of exception(UnboundLocalError) local variable 'local_event' referenced before assignment args=("local variable 'local_event' referenced before assignment",)
```
2022-07-07 11:52:45 -05:00
288 changed files with 5620 additions and 3091 deletions

View File

@@ -69,7 +69,7 @@ with open('pyproject.toml', 'w') as f:
"
python3 -c "$REMOVE_DEV_DEPENDENCIES"
pipx install poetry==1.1.12
pipx install poetry==1.1.14
~/.local/bin/poetry lock
echo "::group::Patched pyproject.toml"

View File

@@ -7,3 +7,4 @@ root = true
[*.py]
indent_style = space
indent_size = 4
max_line_length = 88

View File

@@ -1,3 +1,16 @@
# Commits in this file will be removed from GitHub blame results.
#
# To use this file locally, use:
# git blame --ignore-revs-file="path/to/.git-blame-ignore-revs" <files>
#
# or configure the `blame.ignoreRevsFile` option in your git config.
#
# If ignoring a pull request that was not squash merged, only the merge
# commit needs to be put here. Child commits will be resolved from it.
# Run black (#3679).
8b3d9b6b199abb87246f982d5db356f1966db925
# Black reformatting (#5482).
32e7c9e7f20b57dd081023ac42d6931a8da9b3a3

View File

@@ -328,9 +328,6 @@ jobs:
- arrangement: monolith
database: Postgres
- arrangement: workers
database: Postgres
steps:
- name: Run actions/checkout@v2 for synapse
uses: actions/checkout@v2
@@ -346,6 +343,30 @@ jobs:
shell: bash
name: Run Complement Tests
# XXX When complement with workers is stable, move this back into the standard
# "complement" matrix above.
#
# See https://github.com/matrix-org/synapse/issues/13161
complement-workers:
if: "${{ !failure() && !cancelled() }}"
needs: linting-done
runs-on: ubuntu-latest
steps:
- name: Run actions/checkout@v2 for synapse
uses: actions/checkout@v2
with:
path: synapse
- name: Prepare Complement's Prerequisites
run: synapse/.ci/scripts/setup_complement_prerequisites.sh
- run: |
set -o pipefail
POSTGRES=1 WORKERS=1 COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | gotestfmt
shell: bash
name: Run Complement Tests
# a job which marks all the other jobs as complete, thus allowing PRs to be merged.
tests-done:
if: ${{ always() }}

View File

@@ -127,12 +127,12 @@ jobs:
run: |
set -x
DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx
pipx install poetry==1.1.12
pipx install poetry==1.1.14
poetry remove -n twisted
poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk
poetry lock --no-update
# NOT IN 1.1.12 poetry lock --check
# NOT IN 1.1.14 poetry lock --check
working-directory: synapse
- run: |

View File

@@ -1,3 +1,102 @@
Synapse vNext
=============
As of this release, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. For more information, see the [upgrade notes](https://matrix-org.github.io/synapse/v1.64/upgrade.html#upgrading-to-v1640).
Synapse 1.63.1 (2022-07-20)
===========================
Bugfixes
--------
- Fix a bug introduced in Synapse 1.63.0 where push actions were incorrectly calculated for appservice users. This caused performance issues on servers with large numbers of appservices. ([\#13332](https://github.com/matrix-org/synapse/issues/13332))
Synapse 1.63.0 (2022-07-19)
===========================
Improved Documentation
----------------------
- Clarify that homeserver server names are included in the reported data when the `report_stats` config option is enabled. ([\#13321](https://github.com/matrix-org/synapse/issues/13321))
Synapse 1.63.0rc1 (2022-07-12)
==============================
Features
--------
- Add a rate limit for local users sending invites. ([\#13125](https://github.com/matrix-org/synapse/issues/13125))
- Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of `/publicRooms` by room type. ([\#13031](https://github.com/matrix-org/synapse/issues/13031))
- Improve validation logic in the account data REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
Bugfixes
--------
- Fix a long-standing bug where application services were not able to join remote federated rooms without a profile. ([\#13131](https://github.com/matrix-org/synapse/issues/13131))
- Fix a long-standing bug where `_get_state_map_for_room` might raise errors when third party event rules callbacks are present. ([\#13174](https://github.com/matrix-org/synapse/issues/13174))
- Fix a long-standing bug where the `synapse_port_db` script could fail to copy rows with negative row ids. ([\#13226](https://github.com/matrix-org/synapse/issues/13226))
- Fix a bug introduced in 1.54.0 where appservices would not receive room-less EDUs, like presence, when both [MSC2409](https://github.com/matrix-org/matrix-spec-proposals/pull/2409) and [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202) are enabled. ([\#13236](https://github.com/matrix-org/synapse/issues/13236))
- Fix a bug introduced in 1.62.0 where rows were not deleted from `event_push_actions` table on large servers. ([\#13194](https://github.com/matrix-org/synapse/issues/13194))
- Fix a bug introduced in 1.62.0 where notification counts would get stuck after a highlighted message. ([\#13223](https://github.com/matrix-org/synapse/issues/13223))
- Fix exception when using experimental [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint to look for remote federated imported events before room creation. ([\#13197](https://github.com/matrix-org/synapse/issues/13197))
- Fix [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202)-enabled appservices not receiving to-device messages, preventing messages from being decrypted. ([\#13235](https://github.com/matrix-org/synapse/issues/13235))
Updates to the Docker image
---------------------------
- Bump the version of `lxml` in matrix.org Docker images Debian packages from 4.8.0 to 4.9.1. ([\#13207](https://github.com/matrix-org/synapse/issues/13207))
Improved Documentation
----------------------
- Add an explanation of the `--report-stats` argument to the docs. ([\#13029](https://github.com/matrix-org/synapse/issues/13029))
- Add a helpful example bash script to the contrib directory for creating multiple worker configuration files of the same type. Contributed by @villepeh. ([\#13032](https://github.com/matrix-org/synapse/issues/13032))
- Add missing links to config options. ([\#13166](https://github.com/matrix-org/synapse/issues/13166))
- Add documentation for homeserver usage statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
- Add documentation for the existing `databases` option in the homeserver configuration manual. ([\#13212](https://github.com/matrix-org/synapse/issues/13212))
- Clean up references to sample configuration and redirect users to the configuration manual instead. ([\#13077](https://github.com/matrix-org/synapse/issues/13077), [\#13139](https://github.com/matrix-org/synapse/issues/13139))
- Document how the Synapse team does reviews. ([\#13132](https://github.com/matrix-org/synapse/issues/13132))
- Fix wrong section header for `allow_public_rooms_over_federation` in the homeserver config documentation. ([\#13116](https://github.com/matrix-org/synapse/issues/13116))
Deprecations and Removals
-------------------------
- Remove obsolete and for 8 years unused `RoomEventsStoreTestCase`. Contributed by @arkamar. ([\#13200](https://github.com/matrix-org/synapse/issues/13200))
Internal Changes
----------------
- Add type annotations to `synapse.logging`, `tests.server` and `tests.utils`. ([\#13028](https://github.com/matrix-org/synapse/issues/13028), [\#13103](https://github.com/matrix-org/synapse/issues/13103), [\#13159](https://github.com/matrix-org/synapse/issues/13159), [\#13136](https://github.com/matrix-org/synapse/issues/13136))
- Enforce type annotations for `tests.test_server`. ([\#13135](https://github.com/matrix-org/synapse/issues/13135))
- Support temporary experimental return values for spam checker module callbacks. ([\#13044](https://github.com/matrix-org/synapse/issues/13044))
- Add support to `complement.sh` for skipping the docker build. ([\#13143](https://github.com/matrix-org/synapse/issues/13143), [\#13158](https://github.com/matrix-org/synapse/issues/13158))
- Add support to `complement.sh` for setting the log level using the `SYNAPSE_TEST_LOG_LEVEL` environment variable. ([\#13152](https://github.com/matrix-org/synapse/issues/13152))
- Enable Complement testing in the 'Twisted Trunk' CI runs. ([\#13079](https://github.com/matrix-org/synapse/issues/13079), [\#13157](https://github.com/matrix-org/synapse/issues/13157))
- Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. ([\#13127](https://github.com/matrix-org/synapse/issues/13127))
- Update config used by Complement to allow device name lookup over federation. ([\#13167](https://github.com/matrix-org/synapse/issues/13167))
- Faster room joins: handle race between persisting an event and un-partial stating a room. ([\#13100](https://github.com/matrix-org/synapse/issues/13100))
- Faster room joins: fix race in recalculation of current room state. ([\#13151](https://github.com/matrix-org/synapse/issues/13151))
- Faster room joins: skip waiting for full state when processing incoming events over federation. ([\#13144](https://github.com/matrix-org/synapse/issues/13144))
- Raise a `DependencyError` on missing dependencies instead of a `ConfigError`. ([\#13113](https://github.com/matrix-org/synapse/issues/13113))
- Avoid stripping line breaks from SQL sent to the database. ([\#13129](https://github.com/matrix-org/synapse/issues/13129))
- Apply ratelimiting earlier in processing of `/send` requests. ([\#13134](https://github.com/matrix-org/synapse/issues/13134))
- Improve exception handling when processing events received over federation. ([\#13145](https://github.com/matrix-org/synapse/issues/13145))
- Check that `auto_vacuum` is disabled when porting a SQLite database to Postgres, as `VACUUM`s must not be performed between runs of the script. ([\#13195](https://github.com/matrix-org/synapse/issues/13195))
- Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room. ([\#13119](https://github.com/matrix-org/synapse/issues/13119), [\#13153](https://github.com/matrix-org/synapse/issues/13153))
- Reduce memory consumption when processing incoming events in large rooms. ([\#13078](https://github.com/matrix-org/synapse/issues/13078), [\#13222](https://github.com/matrix-org/synapse/issues/13222))
- Reduce number of queries used to get profile information. Contributed by Nick @ Beeper (@fizzadar). ([\#13209](https://github.com/matrix-org/synapse/issues/13209))
- Reduce number of events queried during room creation. Contributed by Nick @ Beeper (@fizzadar). ([\#13210](https://github.com/matrix-org/synapse/issues/13210))
- More aggressively rotate push actions. ([\#13211](https://github.com/matrix-org/synapse/issues/13211))
- Add `max_line_length` setting for Python files to the `.editorconfig`. Contributed by @sumnerevans @ Beeper. ([\#13228](https://github.com/matrix-org/synapse/issues/13228))
Synapse 1.62.0 (2022-07-05)
===========================
@@ -5,7 +104,6 @@ No significant changes since 1.62.0rc3.
Authors of spam-checker plugins should consult the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.62/docs/upgrade.md#upgrading-to-v1620) to learn about the enriched signatures for spam checker callbacks, which are supported with this release of Synapse.
Synapse 1.62.0rc3 (2022-07-04)
==============================
@@ -45,7 +143,7 @@ Bugfixes
- Update [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786) implementation to check `state_key`. ([\#12939](https://github.com/matrix-org/synapse/issues/12939))
- Fix a bug introduced in Synapse 1.58 where Synapse would not report full version information when installed from a git checkout. This is a best-effort affair and not guaranteed to be stable. ([\#12973](https://github.com/matrix-org/synapse/issues/12973))
- Fix a bug introduced in Synapse 1.60 where Synapse would fail to start if the `sqlite3` module was not available. ([\#12979](https://github.com/matrix-org/synapse/issues/12979))
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
in Synapse v1.41.0. ([\#12991](https://github.com/matrix-org/synapse/issues/12991))
- Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. ([\#13018](https://github.com/matrix-org/synapse/issues/13018))
- Fix a bug introduced in Synapse 1.58 where profile requests for a malformed user ID would ccause an internal error. Synapse now returns 400 Bad Request in this situation. ([\#13041](https://github.com/matrix-org/synapse/issues/13041))

1
changelog.d/12942.misc Normal file
View File

@@ -0,0 +1 @@
Use lower isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper.

1
changelog.d/12943.misc Normal file
View File

@@ -0,0 +1 @@
Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.

View File

@@ -0,0 +1 @@
Drop tables used for groups/communities.

View File

@@ -1 +0,0 @@
Add type annotations to `tests.utils`.

View File

@@ -1 +0,0 @@
Add an explanation of the `--report-stats` argument to the docs.

View File

@@ -1 +0,0 @@
Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of /publicRooms by room type.

View File

@@ -0,0 +1 @@
Provide more info why we don't have any thumbnails to serve.

View File

@@ -1,3 +0,0 @@
Clean up references to sample configuration and redirect users to the configuration manual instead.

View File

@@ -1 +0,0 @@
Enable Complement testing in the 'Twisted Trunk' CI runs.

View File

@@ -1 +0,0 @@
Add documentation for anonymised homeserver statistics collection.

1
changelog.d/13094.misc Normal file
View File

@@ -0,0 +1 @@
Make the AS login method call `Auth.get_user_by_req` for checking the AS token.

View File

@@ -1 +0,0 @@
Faster room joins: Handle race between persisting an event and un-partial stating a room.

View File

@@ -1 +0,0 @@
Add missing type hints to `synapse.logging`.

View File

@@ -1 +0,0 @@
Raise a `DependencyError` on missing dependencies instead of a `ConfigError`.

View File

@@ -1 +0,0 @@
Fix wrong section header for `allow_public_rooms_over_federation` in the homeserver config documentation.

View File

@@ -1 +0,0 @@
Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room.

View File

@@ -1 +0,0 @@
Add a rate limit for local users sending invites.

View File

@@ -1 +0,0 @@
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments.

View File

@@ -1 +0,0 @@
Only one-line SQL statements for logging and tracing.

View File

@@ -1 +0,0 @@
Fix application service not being able to join remote federated room without a profile set.

View File

@@ -1 +0,0 @@
Document how the Synapse team does reviews.

View File

@@ -1 +0,0 @@
Apply ratelimiting earlier in processing of /send request.

View File

@@ -1 +0,0 @@
Enforce type annotations for `tests.test_server`.

View File

@@ -1 +0,0 @@
Add type annotations to `tests.server`.

View File

@@ -1 +0,0 @@
Add a link to the configuration manual from the homeserver sample config documentation.

View File

@@ -1 +0,0 @@
Add support to `complement.sh` for skipping the docker build.

View File

@@ -1 +0,0 @@
Faster joins: skip waiting for full state when processing incoming events over federation.

View File

@@ -1 +0,0 @@
Improve exception handling when processing events received over federation.

View File

@@ -1 +0,0 @@
Improve validation logic in Synapse's REST endpoints.

View File

@@ -1 +0,0 @@
Faster room joins: fix race in recalculation of current room state.

View File

@@ -1 +0,0 @@
Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` environment when using `complement.sh`.

View File

@@ -1 +0,0 @@
Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room.

View File

@@ -1 +0,0 @@
Enable Complement testing in the 'Twisted Trunk' CI runs.

View File

@@ -1 +0,0 @@
Add support to `complement.sh` for skipping the docker build.

View File

@@ -1 +0,0 @@
Improve and fix type hints.

View File

@@ -1 +0,0 @@
Add missing links to config options.

View File

@@ -1 +0,0 @@
Update config used by Complement to allow device name lookup over federation.

1
changelog.d/13172.misc Normal file
View File

@@ -0,0 +1 @@
Always use a version of canonicaljson that supports the C implementation of frozendict.

View File

@@ -1 +0,0 @@
Make use of the more robust `get_current_state` in `_get_state_map_for_room` to avoid breakages.

1
changelog.d/13175.misc Normal file
View File

@@ -0,0 +1 @@
Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper.

View File

@@ -0,0 +1 @@
Drop support for delegating email verification to an external server.

View File

@@ -1 +0,0 @@
Fix bug where rows were not deleted from `event_push_actions` table on large servers. Introduced in v1.62.0.

View File

@@ -1 +0,0 @@
Check that `auto_vacuum` is disabled when porting a SQLite database to Postgres, as `VACUUM`s must not be performed between runs of the script.

1
changelog.d/13198.misc Normal file
View File

@@ -0,0 +1 @@
Refactor receipts servlet logic to avoid duplicated code.

View File

@@ -1 +0,0 @@
Remove obsolete and for 8 years unused `RoomEventsStoreTestCase`. Contributed by @arkamar.

View File

@@ -0,0 +1 @@
Allow pagination from remote event after discovering it from MSC3030 `/timestamp_to_event`.

View File

@@ -1 +0,0 @@
Bump the version of `lxml` in matrix.org Docker images Debian packages from 4.8.0 to 4.9.1.

View File

@@ -0,0 +1 @@
Add a `room_type` field in the responses for the list room and room details admin API. Contributed by @andrewdoh.

View File

@@ -1 +0,0 @@
Reduce number of queries used to get profile information. Contributed by Nick @ Beeper (@fizzadar).

1
changelog.d/13215.misc Normal file
View File

@@ -0,0 +1 @@
Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table.

1
changelog.d/13218.misc Normal file
View File

@@ -0,0 +1 @@
Remove unused database table `event_reference_hashes`.

View File

@@ -0,0 +1 @@
Add support for room version 10.

1
changelog.d/13224.misc Normal file
View File

@@ -0,0 +1 @@
Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@fizzadar).

1
changelog.d/13231.doc Normal file
View File

@@ -0,0 +1 @@
Provide an example of using the Admin API. Contributed by @jejo86.

1
changelog.d/13233.doc Normal file
View File

@@ -0,0 +1 @@
Move the documentation for how URL previews work to the URL preview module.

View File

@@ -0,0 +1 @@
Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.

1
changelog.d/13240.misc Normal file
View File

@@ -0,0 +1 @@
Call the v2 identity service `/3pid/unbind` endpoint, rather than v1.

1
changelog.d/13242.misc Normal file
View File

@@ -0,0 +1 @@
Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).

1
changelog.d/13251.misc Normal file
View File

@@ -0,0 +1 @@
Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@fizzadar).

1
changelog.d/13253.misc Normal file
View File

@@ -0,0 +1 @@
Preparatory work for a per-room rate limiter on joins.

1
changelog.d/13254.misc Normal file
View File

@@ -0,0 +1 @@
Preparatory work for a per-room rate limiter on joins.

1
changelog.d/13255.misc Normal file
View File

@@ -0,0 +1 @@
Preparatory work for a per-room rate limiter on joins.

1
changelog.d/13257.misc Normal file
View File

@@ -0,0 +1 @@
Log the stack when waiting for an entire room to be un-partial stated.

1
changelog.d/13258.misc Normal file
View File

@@ -0,0 +1 @@
Fix spurious warning when fetching state after a missing prev event.

1
changelog.d/13260.misc Normal file
View File

@@ -0,0 +1 @@
Clean-up tests for notifications.

1
changelog.d/13261.doc Normal file
View File

@@ -0,0 +1 @@
Move the documentation for how URL previews work to the URL preview module.

1
changelog.d/13263.bugfix Normal file
View File

@@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the "enable_email_notifs" and "email_notifs_for_new_users" options were enabled. Contributed by @thomasweston12.

1
changelog.d/13266.misc Normal file
View File

@@ -0,0 +1 @@
Do not fail build if complement with workers fails.

1
changelog.d/13267.misc Normal file
View File

@@ -0,0 +1 @@
Don't pull out state in `compute_event_context` for unconflicted state.

1
changelog.d/13270.bugfix Normal file
View File

@@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.40 where a user invited to a restricted room would be briefly unable to join.

1
changelog.d/13271.doc Normal file
View File

@@ -0,0 +1 @@
Add another `contrib` script to help set up worker processes. Contributed by @villepeh.

1
changelog.d/13274.misc Normal file
View File

@@ -0,0 +1 @@
Don't pull out state in `compute_event_context` for unconflicted state.

View File

@@ -0,0 +1 @@
Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttle additional joins if that rate grows too large.

1
changelog.d/13278.bugfix Normal file
View File

@@ -0,0 +1 @@
Fix long-standing bug where in rare instances Synapse could store the incorrect state for a room after a state resolution.

1
changelog.d/13279.misc Normal file
View File

@@ -0,0 +1 @@
Reduce the rebuild time for the complement-synapse docker image.

1
changelog.d/13281.misc Normal file
View File

@@ -0,0 +1 @@
Don't pull out the full state when creating an event.

1
changelog.d/13284.misc Normal file
View File

@@ -0,0 +1 @@
Update locked version of `frozendict` to 2.3.2, which has a fix for a memory leak.

1
changelog.d/13285.misc Normal file
View File

@@ -0,0 +1 @@
Upgrade from Poetry 1.1.14 to 1.1.12, to fix bugs when locking packages.

1
changelog.d/13292.misc Normal file
View File

@@ -0,0 +1 @@
Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently.

1
changelog.d/13296.bugfix Normal file
View File

@@ -0,0 +1 @@
Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced.

1
changelog.d/13297.misc Normal file
View File

@@ -0,0 +1 @@
Use `HTTPStatus` constants in place of literals in tests.

1
changelog.d/13299.misc Normal file
View File

@@ -0,0 +1 @@
Improve performance of query `_get_subset_users_in_room_with_profiles`.

1
changelog.d/13300.misc Normal file
View File

@@ -0,0 +1 @@
Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`.

1
changelog.d/13303.misc Normal file
View File

@@ -0,0 +1 @@
Remove unnecessary `json.dumps` from tests.

1
changelog.d/13307.misc Normal file
View File

@@ -0,0 +1 @@
Don't pull out the full state when creating an event.

1
changelog.d/13308.misc Normal file
View File

@@ -0,0 +1 @@
Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).

1
changelog.d/13310.misc Normal file
View File

@@ -0,0 +1 @@
Reduce memory usage of sending dummy events.

1
changelog.d/13311.misc Normal file
View File

@@ -0,0 +1 @@
Prevent formatting changes of [#3679](https://github.com/matrix-org/synapse/pull/3679) from appearing in `git blame`.

1
changelog.d/13314.doc Normal file
View File

@@ -0,0 +1 @@
Add notes when config options where changed. Contributed by @behrmann.

1
changelog.d/13318.misc Normal file
View File

@@ -0,0 +1 @@
Validate federation destinations and log an error if a destination is invalid.

1
changelog.d/13320.misc Normal file
View File

@@ -0,0 +1 @@
Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation.

1
changelog.d/13323.misc Normal file
View File

@@ -0,0 +1 @@
Reduce memory usage of state caches.

1
changelog.d/13324.misc Normal file
View File

@@ -0,0 +1 @@
Reduce the amount of state we store in the `state_cache`.

View File

@@ -0,0 +1 @@
Stop builindg `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life.

Some files were not shown because too many files have changed in this diff Show More