kaslo/synapse
1
0
Fork 0
Code Wiki Activity

Merge commit '4a54b821b' into anoa/dinsic_release_1_23_1

Browse Source
This commit is contained in:
Andrew Morgan
2020-12-31 14:35:50 +00:00
parent 883c742960 4a54b821bb
commit ca151ff0bd
74 changed files with 539 additions and 285 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.buildkite/test_db.db
BIN
View File
Binary file not shown.
CHANGES.md
+86
View File
@@ -1,3 +1,89 @@
Synapse 1.23.0rc1 (2020-11-13)
==============================
Features
--------
- Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](https://github.com/matrix-org/synapse/issues/8286))
- Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](https://github.com/matrix-org/synapse/issues/8519))
- Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](https://github.com/matrix-org/synapse/issues/8539))
- Support generating structured logs via the standard logging configuration. ([\#8607](https://github.com/matrix-org/synapse/issues/8607), [\#8685](https://github.com/matrix-org/synapse/issues/8685))
- Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](https://github.com/matrix-org/synapse/issues/8610), [\#8689](https://github.com/matrix-org/synapse/issues/8689))
- Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](https://github.com/matrix-org/synapse/issues/8647))
- Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](https://github.com/matrix-org/synapse/issues/8700))
- Add `displayname` to Shared-Secret Registration for admins. ([\#8722](https://github.com/matrix-org/synapse/issues/8722))
Bugfixes
--------
- Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already. ([\#8455](https://github.com/matrix-org/synapse/issues/8455))
- Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information. ([\#8580](https://github.com/matrix-org/synapse/issues/8580))
- Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error. ([\#8620](https://github.com/matrix-org/synapse/issues/8620))
- Fix email notifications for invites without local state. ([\#8627](https://github.com/matrix-org/synapse/issues/8627))
- Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500. ([\#8628](https://github.com/matrix-org/synapse/issues/8628))
- Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](https://github.com/matrix-org/synapse/issues/8632))
- Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](https://github.com/matrix-org/synapse/issues/8643))
- Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](https://github.com/matrix-org/synapse/issues/8682))
- Fix bug where Synapse would not recover after losing connection to the database. ([\#8726](https://github.com/matrix-org/synapse/issues/8726))
- Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](https://github.com/matrix-org/synapse/issues/8728))
- Fix port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](https://github.com/matrix-org/synapse/issues/8729))
- Fix port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](https://github.com/matrix-org/synapse/issues/8730))
- Fix port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](https://github.com/matrix-org/synapse/issues/8755))
Improved Documentation
----------------------
- Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk. ([\#8582](https://github.com/matrix-org/synapse/issues/8582))
- Improve the sample configuration for single sign-on providers. ([\#8635](https://github.com/matrix-org/synapse/issues/8635))
- Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs. ([\#8657](https://github.com/matrix-org/synapse/issues/8657))
- Note support for Python 3.9. ([\#8665](https://github.com/matrix-org/synapse/issues/8665))
- Minor updates to docs on running tests. ([\#8666](https://github.com/matrix-org/synapse/issues/8666))
- Interlink prometheus/grafana documentation. ([\#8667](https://github.com/matrix-org/synapse/issues/8667))
- Notes on SSO logins and media_repository worker. ([\#8701](https://github.com/matrix-org/synapse/issues/8701))
- Document experimental support for running multiple event persisters. ([\#8706](https://github.com/matrix-org/synapse/issues/8706))
- Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`. ([\#8714](https://github.com/matrix-org/synapse/issues/8714))
- Migrate documentation `docs/admin_api/event_reports` to markdown. ([\#8742](https://github.com/matrix-org/synapse/issues/8742))
- Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95. ([\#8746](https://github.com/matrix-org/synapse/issues/8746))
Internal Changes
----------------
- Optimise `/createRoom` with multiple invited users. ([\#8559](https://github.com/matrix-org/synapse/issues/8559))
- Implement and use an @lru_cache decorator. ([\#8595](https://github.com/matrix-org/synapse/issues/8595))
- Don't instansiate Requester directly. ([\#8614](https://github.com/matrix-org/synapse/issues/8614))
- Type hints for `RegistrationStore`. ([\#8615](https://github.com/matrix-org/synapse/issues/8615))
- Change schema to support access tokens belonging to one user but granting access to another. ([\#8616](https://github.com/matrix-org/synapse/issues/8616))
- Remove unused OPTIONS handlers. ([\#8621](https://github.com/matrix-org/synapse/issues/8621))
- Run `mypy` as part of the lint.sh script. ([\#8633](https://github.com/matrix-org/synapse/issues/8633))
- Correct Synapse's PyPI package name in the OpenID Connect installation instructions. ([\#8634](https://github.com/matrix-org/synapse/issues/8634))
- Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg. ([\#8636](https://github.com/matrix-org/synapse/issues/8636))
- Fix typos and spelling errors in the code. ([\#8639](https://github.com/matrix-org/synapse/issues/8639))
- Reduce number of OpenTracing spans started. ([\#8640](https://github.com/matrix-org/synapse/issues/8640), [\#8668](https://github.com/matrix-org/synapse/issues/8668), [\#8670](https://github.com/matrix-org/synapse/issues/8670))
- Add field `total` to device list in admin API. ([\#8644](https://github.com/matrix-org/synapse/issues/8644))
- Add more type hints to the application services code. ([\#8655](https://github.com/matrix-org/synapse/issues/8655), [\#8693](https://github.com/matrix-org/synapse/issues/8693))
- Tell Black to format code for Python 3.5. ([\#8664](https://github.com/matrix-org/synapse/issues/8664))
- Don't pull event from DB when handling replication traffic. ([\#8669](https://github.com/matrix-org/synapse/issues/8669))
- Abstract some invite-related code in preparation for landing knocking. ([\#8671](https://github.com/matrix-org/synapse/issues/8671), [\#8688](https://github.com/matrix-org/synapse/issues/8688))
- Clarify representation of events in logfiles. ([\#8679](https://github.com/matrix-org/synapse/issues/8679))
- Don't require `hiredis` package to be installed to run unit tests. ([\#8680](https://github.com/matrix-org/synapse/issues/8680))
- Fix typing info on cache call signature to accept `on_invalidate`. ([\#8684](https://github.com/matrix-org/synapse/issues/8684))
- Fail tests if they do not await coroutines. ([\#8690](https://github.com/matrix-org/synapse/issues/8690))
- Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`. ([\#8694](https://github.com/matrix-org/synapse/issues/8694))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8697](https://github.com/matrix-org/synapse/issues/8697))
- Use Python 3.8 in Docker images by default. ([\#8698](https://github.com/matrix-org/synapse/issues/8698))
- Remove the "draft" status of the Room Details Admin API. ([\#8702](https://github.com/matrix-org/synapse/issues/8702))
- Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. ([\#8705](https://github.com/matrix-org/synapse/issues/8705))
- Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. ([\#8708](https://github.com/matrix-org/synapse/issues/8708))
- Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests. ([\#8712](https://github.com/matrix-org/synapse/issues/8712))
- Consolidate duplicated lists of purged tables that are checked in tests. ([\#8713](https://github.com/matrix-org/synapse/issues/8713))
- Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config. ([\#8718](https://github.com/matrix-org/synapse/issues/8718))
- Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request. ([\#8719](https://github.com/matrix-org/synapse/issues/8719))
- Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. ([\#8752](https://github.com/matrix-org/synapse/issues/8752))
Synapse 1.22.1 (2020-10-30)
===========================
README.rst
+10 -6
View File
@@ -261,18 +261,22 @@ to install using pip and a virtualenv::
pip install -e ".[all,test]"
This will run a process of downloading and installing all the needed
dependencies into a virtual env.
dependencies into a virtual env. If any dependencies fail to install,
try installing the failing modules individually::
Once this is done, you may wish to run Synapse's unit tests, to
check that everything is installed as it should be::
pip install -e "module-name"
Once this is done, you may wish to run Synapse's unit tests to
check that everything is installed correctly::
python -m twisted.trial tests
This should end with a 'PASSED' result::
This should end with a 'PASSED' result (note that exact numbers will
differ)::
Ran 1266 tests in 643.930s
Ran 1337 tests in 716.064s
PASSED (skips=15, successes=1251)
PASSED (skips=15, successes=1322)
Running the Integration Tests
=============================
changelog.d/8286.feature
-1
View File
@@ -1 +0,0 @@
Add a push rule that highlights when a jitsi conference is created in a room.
changelog.d/8455.bugfix
-1
View File
@@ -1 +0,0 @@
Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already.
changelog.d/8519.feature
-1
View File
@@ -1 +0,0 @@
Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel.
changelog.d/8539.feature
-1
View File
@@ -1 +0,0 @@
Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel.
changelog.d/8559.misc
-1
View File
@@ -1 +0,0 @@
Optimise `/createRoom` with multiple invited users.
changelog.d/8580.bugfix
-1
View File
@@ -1 +0,0 @@
Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information.
changelog.d/8582.doc
-1
View File
@@ -1 +0,0 @@
Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk.
changelog.d/8595.misc
-1
View File
@@ -1 +0,0 @@
Implement and use an @lru_cache decorator.
changelog.d/8607.feature
-1
View File
@@ -1 +0,0 @@
Support generating structured logs via the standard logging configuration.
changelog.d/8610.feature
-1
View File
@@ -1 +0,0 @@
Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.
changelog.d/8614.misc
-1
View File
@@ -1 +0,0 @@
Don't instansiate Requester directly.
changelog.d/8615.misc
-1
View File
@@ -1 +0,0 @@
Type hints for `RegistrationStore`.
changelog.d/8616.misc
-1
View File
@@ -1 +0,0 @@
Change schema to support access tokens belonging to one user but granting access to another.
changelog.d/8620.bugfix
-1
View File
@@ -1 +0,0 @@
Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error.
changelog.d/8621.misc
-1
View File
@@ -1 +0,0 @@
Remove unused OPTIONS handlers.
changelog.d/8627.bugfix
-1
View File
@@ -1 +0,0 @@
Fix email notifications for invites without local state.
changelog.d/8628.bugfix
-1
View File
@@ -1 +0,0 @@
Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500.
changelog.d/8632.bugfix
-1
View File
@@ -1 +0,0 @@
Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded.
changelog.d/8633.misc
-1
View File
@@ -1 +0,0 @@
Run `mypy` as part of the lint.sh script.
changelog.d/8634.misc
-1
View File
@@ -1 +0,0 @@
Correct Synapse's PyPI package name in the OpenID Connect installation instructions.
changelog.d/8635.doc
-1
View File
@@ -1 +0,0 @@
Improve the sample configuration for single sign-on providers.
changelog.d/8636.misc
-1
View File
@@ -1 +0,0 @@
Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg.
changelog.d/8639.misc
-1
View File
@@ -1 +0,0 @@
Fix typos and spelling errors in the code.
changelog.d/8640.misc
-1
View File
@@ -1 +0,0 @@
Reduce number of OpenTracing spans started.
changelog.d/8643.bugfix
-1
View File
@@ -1 +0,0 @@
Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0.
changelog.d/8644.misc
-1
View File
@@ -1 +0,0 @@
Add field `total` to device list in admin API.
changelog.d/8647.feature
-1
View File
@@ -1 +0,0 @@
Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel.
changelog.d/8655.misc
-1
View File
@@ -1 +0,0 @@
Add more type hints to the application services code.
changelog.d/8657.doc
-1
View File
@@ -1 +0,0 @@
Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs.
changelog.d/8664.misc
-1
View File
@@ -1 +0,0 @@
Tell Black to format code for Python 3.5.
changelog.d/8665.doc
-1
View File
@@ -1 +0,0 @@
Note support for Python 3.9.
changelog.d/8666.doc
-1
View File
@@ -1 +0,0 @@
Minor updates to docs on running tests.
changelog.d/8667.doc
-1
View File
@@ -1 +0,0 @@
Interlink prometheus/grafana documentation.
changelog.d/8668.misc
-1
View File
@@ -1 +0,0 @@
Reduce number of OpenTracing spans started.
changelog.d/8669.misc
-1
View File
@@ -1 +0,0 @@
Don't pull event from DB when handling replication traffic.
changelog.d/8670.misc
-1
View File
@@ -1 +0,0 @@
Reduce number of OpenTracing spans started.
changelog.d/8671.misc
-1
View File
@@ -1 +0,0 @@
Abstract some invite-related code in preparation for landing knocking.
changelog.d/8679.misc
-1
View File
@@ -1 +0,0 @@
Clarify representation of events in logfiles.
changelog.d/8680.misc
-1
View File
@@ -1 +0,0 @@
Don't require `hiredis` package to be installed to run unit tests.
changelog.d/8682.bugfix
-1
View File
@@ -1 +0,0 @@
Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories.
changelog.d/8684.misc
-1
View File
@@ -1 +0,0 @@
Fix typing info on cache call signature to accept `on_invalidate`.
changelog.d/8685.feature
-1
View File
@@ -1 +0,0 @@
Support generating structured logs via the standard logging configuration.
changelog.d/8688.misc
-1
View File
@@ -1 +0,0 @@
Abstract some invite-related code in preparation for landing knocking.
changelog.d/8689.feature
-1
View File
@@ -1 +0,0 @@
Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.
changelog.d/8690.misc
-1
View File
@@ -1 +0,0 @@
Fail tests if they do not await coroutines.
changelog.d/8693.misc
-1
View File
@@ -1 +0,0 @@
Add more type hints to the application services code.
changelog.d/8694.misc
-1
View File
@@ -1 +0,0 @@
Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`.
changelog.d/8697.misc
-1
View File
@@ -1 +0,0 @@
Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting.
changelog.d/8698.misc
-1
View File
@@ -1 +0,0 @@
Use Python 3.8 in Docker images by default.
changelog.d/8700.feature
-1
View File
@@ -1 +0,0 @@
Add an admin API for local user media statistics. Contributed by @dklimpel.
changelog.d/8701.doc
-1
View File
@@ -1 +0,0 @@
Notes on SSO logins and media_repository worker.
changelog.d/8702.misc
-1
View File
@@ -1 +0,0 @@
Remove the "draft" status of the Room Details Admin API.
changelog.d/8705.misc
-1
View File
@@ -1 +0,0 @@
Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile.
changelog.d/8706.doc
-1
View File
@@ -1 +0,0 @@
Document experimental support for running multiple event persisters.
changelog.d/8708.misc
-1
View File
@@ -1 +0,0 @@
Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room.
changelog.d/8713.misc
-1
View File
@@ -1 +0,0 @@
Consolidate duplicated lists of purged tables that are checked in tests.
changelog.d/8714.doc
-1
View File
@@ -1 +0,0 @@
Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`.
changelog.d/8719.misc
-1
View File
@@ -1 +0,0 @@
Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request.
changelog.d/8722.feature
-1
View File
@@ -1 +0,0 @@
Add `displayname` to Shared-Secret Registration for admins.
changelog.d/8728.bugfix
-1
View File
@@ -1 +0,0 @@
Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms.
docs/admin_api/event_reports.md
+172
View File
@@ -0,0 +1,172 @@
# Show reported events
This API returns information about reported events.
The api is:
```
GET /_synapse/admin/v1/event_reports?from=0&limit=10
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [README.rst](README.rst).
It returns a JSON body like the following:
```json
{
"event_reports": [
{
"event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
"id": 2,
"reason": "foo",
"score": -100,
"received_ts": 1570897107409,
"canonical_alias": "#alias1:matrix.org",
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"name": "Matrix HQ",
"sender": "@foobar:matrix.org",
"user_id": "@foo:matrix.org"
},
{
"event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
"id": 3,
"reason": "bar",
"score": -100,
"received_ts": 1598889612059,
"canonical_alias": "#alias2:matrix.org",
"room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
"name": "Your room name here",
"sender": "@foobar:matrix.org",
"user_id": "@bar:matrix.org"
}
],
"next_token": 2,
"total": 4
}
```
To paginate, check for `next_token` and if present, call the endpoint again with `from`
set to the value of `next_token`. This will return a new page.
If the endpoint does not return a `next_token` then there are no more reports to
paginate through.
**URL parameters:**
* `limit`: integer - Is optional but is used for pagination, denoting the maximum number
of items to return in this call. Defaults to `100`.
* `from`: integer - Is optional but used for pagination, denoting the offset in the
returned results. This should be treated as an opaque value and not explicitly set to
anything other than the return value of `next_token` from a previous call. Defaults to `0`.
* `dir`: string - Direction of event report order. Whether to fetch the most recent
first (`b`) or the oldest first (`f`). Defaults to `b`.
* `user_id`: string - Is optional and filters to only return users with user IDs that
contain this value. This is the user who reported the event and wrote the reason.
* `room_id`: string - Is optional and filters to only return rooms with room IDs that
contain this value.
**Response**
The following fields are returned in the JSON response body:
* `id`: integer - ID of event report.
* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
report was sent.
* `room_id`: string - The ID of the room in which the event being reported is located.
* `name`: string - The name of the room.
* `event_id`: string - The ID of the reported event.
* `user_id`: string - This is the user who reported the event and wrote the reason.
* `reason`: string - Comment made by the `user_id` in this report. May be blank.
* `score`: integer - Content is reported based upon a negative score, where -100 is
"most offensive" and 0 is "inoffensive".
* `sender`: string - This is the ID of the user who sent the original message/event that
was reported.
* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
have a canonical alias set.
* `next_token`: integer - Indication for pagination. See above.
* `total`: integer - Total number of event reports related to the query
(`user_id` and `room_id`).
# Show details of a specific event report
This API returns information about a specific event report.
The api is:
```
GET /_synapse/admin/v1/event_reports/<report_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [README.rst](README.rst).
It returns a JSON body like the following:
```jsonc
{
"event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
"event_json": {
"auth_events": [
"$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
"$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
],
"content": {
"body": "matrix.org: This Week in Matrix",
"format": "org.matrix.custom.html",
"formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
"msgtype": "m.notice"
},
"depth": 546,
"hashes": {
"sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
},
"origin": "matrix.org",
"origin_server_ts": 1592291711430,
"prev_events": [
"$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
],
"prev_state": [],
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"sender": "@foobar:matrix.org",
"signatures": {
"matrix.org": {
"ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
}
},
"type": "m.room.message",
"unsigned": {
"age_ts": 1592291711430,
}
},
"id": <report_id>,
"reason": "foo",
"score": -100,
"received_ts": 1570897107409,
"canonical_alias": "#alias1:matrix.org",
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"name": "Matrix HQ",
"sender": "@foobar:matrix.org",
"user_id": "@foo:matrix.org"
}
```
**URL parameters:**
* `report_id`: string - The ID of the event report.
**Response**
The following fields are returned in the JSON response body:
* `id`: integer - ID of event report.
* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
report was sent.
* `room_id`: string - The ID of the room in which the event being reported is located.
* `name`: string - The name of the room.
* `event_id`: string - The ID of the reported event.
* `user_id`: string - This is the user who reported the event and wrote the reason.
* `reason`: string - Comment made by the `user_id` in this report. May be blank.
* `score`: integer - Content is reported based upon a negative score, where -100 is
"most offensive" and 0 is "inoffensive".
* `sender`: string - This is the ID of the user who sent the original message/event that
was reported.
* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
have a canonical alias set.
* `event_json`: object - Details of the original event that was reported.
docs/admin_api/event_reports.rst
-165
View File
@@ -1,165 +0,0 @@
Show reported events
====================
This API returns information about reported events.
The api is::
GET /_synapse/admin/v1/event_reports?from=0&limit=10
To use it, you will need to authenticate by providing an ``access_token`` for a
server admin: see `README.rst <README.rst>`_.
It returns a JSON body like the following:
.. code:: jsonc
{
"event_reports": [
{
"event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
"id": 2,
"reason": "foo",
"score": -100,
"received_ts": 1570897107409,
"canonical_alias": "#alias1:matrix.org",
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"name": "Matrix HQ",
"sender": "@foobar:matrix.org",
"user_id": "@foo:matrix.org"
},
{
"event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
"id": 3,
"reason": "bar",
"score": -100,
"received_ts": 1598889612059,
"canonical_alias": "#alias2:matrix.org",
"room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
"name": "Your room name here",
"sender": "@foobar:matrix.org",
"user_id": "@bar:matrix.org"
}
],
"next_token": 2,
"total": 4
}
To paginate, check for ``next_token`` and if present, call the endpoint again
with ``from`` set to the value of ``next_token``. This will return a new page.
If the endpoint does not return a ``next_token`` then there are no more
reports to paginate through.
**URL parameters:**
- ``limit``: integer - Is optional but is used for pagination,
denoting the maximum number of items to return in this call. Defaults to ``100``.
- ``from``: integer - Is optional but used for pagination,
denoting the offset in the returned results. This should be treated as an opaque value and
not explicitly set to anything other than the return value of ``next_token`` from a previous call.
Defaults to ``0``.
- ``dir``: string - Direction of event report order. Whether to fetch the most recent first (``b``) or the
oldest first (``f``). Defaults to ``b``.
- ``user_id``: string - Is optional and filters to only return users with user IDs that contain this value.
This is the user who reported the event and wrote the reason.
- ``room_id``: string - Is optional and filters to only return rooms with room IDs that contain this value.
**Response**
The following fields are returned in the JSON response body:
- ``id``: integer - ID of event report.
- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
- ``room_id``: string - The ID of the room in which the event being reported is located.
- ``name``: string - The name of the room.
- ``event_id``: string - The ID of the reported event.
- ``user_id``: string - This is the user who reported the event and wrote the reason.
- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
- ``next_token``: integer - Indication for pagination. See above.
- ``total``: integer - Total number of event reports related to the query (``user_id`` and ``room_id``).
Show details of a specific event report
=======================================
This API returns information about a specific event report.
The api is::
GET /_synapse/admin/v1/event_reports/<report_id>
To use it, you will need to authenticate by providing an ``access_token`` for a
server admin: see `README.rst <README.rst>`_.
It returns a JSON body like the following:
.. code:: jsonc
{
"event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
"event_json": {
"auth_events": [
"$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
"$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
],
"content": {
"body": "matrix.org: This Week in Matrix",
"format": "org.matrix.custom.html",
"formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
"msgtype": "m.notice"
},
"depth": 546,
"hashes": {
"sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
},
"origin": "matrix.org",
"origin_server_ts": 1592291711430,
"prev_events": [
"$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
],
"prev_state": [],
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"sender": "@foobar:matrix.org",
"signatures": {
"matrix.org": {
"ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
}
},
"type": "m.room.message",
"unsigned": {
"age_ts": 1592291711430,
}
},
"id": <report_id>,
"reason": "foo",
"score": -100,
"received_ts": 1570897107409,
"canonical_alias": "#alias1:matrix.org",
"room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
"name": "Matrix HQ",
"sender": "@foobar:matrix.org",
"user_id": "@foo:matrix.org"
}
**URL parameters:**
- ``report_id``: string - The ID of the event report.
**Response**
The following fields are returned in the JSON response body:
- ``id``: integer - ID of event report.
- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
- ``room_id``: string - The ID of the room in which the event being reported is located.
- ``name``: string - The name of the room.
- ``event_id``: string - The ID of the reported event.
- ``user_id``: string - This is the user who reported the event and wrote the reason.
- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
- ``event_json``: object - Details of the original event that was reported.
docs/sample_config.yaml
+22
View File
@@ -1738,6 +1738,28 @@ saml2_config:
#description: ["My awesome SP", "en"]
#name: ["Test SP", "en"]
#ui_info:
# display_name:
# - lang: en
# text: "Display Name is the descriptive name of your service."
# description:
# - lang: en
# text: "Description should be a short paragraph explaining the purpose of the service."
# information_url:
# - lang: en
# text: "https://example.com/terms-of-service"
# privacy_statement_url:
# - lang: en
# text: "https://example.com/privacy-policy"
# keywords:
# - lang: en
# text: ["Matrix", "Element"]
# logo:
# - lang: en
# text: "https://example.com/logo.svg"
# width: "200"
# height: "80"
#organization:
# name: Example com
# display_name:
scripts/synapse_port_db
+118 -40
View File
@@ -22,7 +22,7 @@ import logging
import sys
import time
import traceback
from typing import Optional
from typing import Dict, Optional, Set
import yaml
@@ -294,6 +294,34 @@ class Porter(object):
return table, already_ported, total_to_port, forward_chunk, backward_chunk
async def get_table_constraints(self) -> Dict[str, Set[str]]:
"""Returns a map of tables that have foreign key constraints to tables they depend on.
"""
def _get_constraints(txn):
# We can pull the information about foreign key constraints out from
# the postgres schema tables.
sql = """
SELECT DISTINCT
tc.table_name,
ccu.table_name AS foreign_table_name
FROM
information_schema.table_constraints AS tc
INNER JOIN information_schema.constraint_column_usage AS ccu
USING (table_schema, constraint_name)
WHERE tc.constraint_type = 'FOREIGN KEY';
"""
txn.execute(sql)
results = {}
for table, foreign_table in txn:
results.setdefault(table, set()).add(foreign_table)
return results
return await self.postgres_store.db_pool.runInteraction(
"get_table_constraints", _get_constraints
)
async def handle_table(
self, table, postgres_size, table_size, forward_chunk, backward_chunk
):
@@ -593,7 +621,18 @@ class Porter(object):
"create_port_table", create_port_table
)
# Step 2. Get tables.
# Step 2. Set up sequences
#
# We do this before porting the tables so that event if we fail half
# way through the postgres DB always have sequences that are greater
# than their respective tables. If we don't then creating the
# `DataStore` object will fail due to the inconsistency.
self.progress.set_state("Setting up sequence generators")
await self._setup_state_group_id_seq()
await self._setup_user_id_seq()
await self._setup_events_stream_seqs()
# Step 3. Get tables.
self.progress.set_state("Fetching tables")
sqlite_tables = await self.sqlite_store.db_pool.simple_select_onecol(
table="sqlite_master", keyvalues={"type": "table"}, retcol="name"
@@ -608,7 +647,7 @@ class Porter(object):
tables = set(sqlite_tables) & set(postgres_tables)
logger.info("Found %d tables", len(tables))
# Step 3. Figure out what still needs copying
# Step 4. Figure out what still needs copying
self.progress.set_state("Checking on port progress")
setup_res = await make_deferred_yieldable(
defer.gatherResults(
@@ -621,21 +660,43 @@ class Porter(object):
consumeErrors=True,
)
)
# Map from table name to args passed to `handle_table`, i.e. a tuple
# of: `postgres_size`, `table_size`, `forward_chunk`, `backward_chunk`.
tables_to_port_info_map = {r[0]: r[1:] for r in setup_res}
# Step 4. Do the copying.
# Step 5. Do the copying.
#
# This is slightly convoluted as we need to ensure tables are ported
# in the correct order due to foreign key constraints.
self.progress.set_state("Copying to postgres")
await make_deferred_yieldable(
defer.gatherResults(
[run_in_background(self.handle_table, *res) for res in setup_res],
consumeErrors=True,
)
)
# Step 5. Set up sequences
self.progress.set_state("Setting up sequence generators")
await self._setup_state_group_id_seq()
await self._setup_user_id_seq()
await self._setup_events_stream_seqs()
constraints = await self.get_table_constraints()
tables_ported = set() # type: Set[str]
while tables_to_port_info_map:
# Pulls out all tables that are still to be ported and which
# only depend on tables that are already ported (if any).
tables_to_port = [
table
for table in tables_to_port_info_map
if not constraints.get(table, set()) - tables_ported
]
await make_deferred_yieldable(
defer.gatherResults(
[
run_in_background(
self.handle_table,
table,
*tables_to_port_info_map.pop(table),
)
for table in tables_to_port
],
consumeErrors=True,
)
)
tables_ported.update(tables_to_port)
self.progress.done()
except Exception as e:
@@ -794,45 +855,62 @@ class Porter(object):
return done, remaining + done
def _setup_state_group_id_seq(self):
async def _setup_state_group_id_seq(self):
curr_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
table="state_groups", keyvalues={}, retcol="MAX(id)", allow_none=True
)
if not curr_id:
return
def r(txn):
txn.execute("SELECT MAX(id) FROM state_groups")
curr_id = txn.fetchone()[0]
if not curr_id:
return
next_id = curr_id + 1
txn.execute("ALTER SEQUENCE state_group_id_seq RESTART WITH %s", (next_id,))
return self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
await self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
async def _setup_user_id_seq(self):
curr_id = await self.sqlite_store.db_pool.runInteraction(
"setup_user_id_seq", find_max_generated_user_id_localpart
)
def _setup_user_id_seq(self):
def r(txn):
next_id = find_max_generated_user_id_localpart(txn) + 1
next_id = curr_id + 1
txn.execute("ALTER SEQUENCE user_id_seq RESTART WITH %s", (next_id,))
return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
def _setup_events_stream_seqs(self):
def r(txn):
txn.execute("SELECT MAX(stream_ordering) FROM events")
curr_id = txn.fetchone()[0]
if curr_id:
next_id = curr_id + 1
async def _setup_events_stream_seqs(self):
"""Set the event stream sequences to the correct values.
"""
# We get called before we've ported the events table, so we need to
# fetch the current positions from the SQLite store.
curr_forward_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
table="events", keyvalues={}, retcol="MAX(stream_ordering)", allow_none=True
)
curr_backward_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
table="events",
keyvalues={},
retcol="MAX(-MIN(stream_ordering), 1)",
allow_none=True,
)
def _setup_events_stream_seqs_set_pos(txn):
if curr_forward_id:
txn.execute(
"ALTER SEQUENCE events_stream_seq RESTART WITH %s", (next_id,)
"ALTER SEQUENCE events_stream_seq RESTART WITH %s",
(curr_forward_id + 1,),
)
txn.execute("SELECT -MIN(stream_ordering) FROM events")
curr_id = txn.fetchone()[0]
if curr_id:
next_id = curr_id + 1
txn.execute(
"ALTER SEQUENCE events_backfill_stream_seq RESTART WITH %s",
(next_id,),
)
txn.execute(
"ALTER SEQUENCE events_backfill_stream_seq RESTART WITH %s",
(curr_backward_id + 1,),
)
return self.postgres_store.db_pool.runInteraction(
"_setup_events_stream_seqs", r
return await self.postgres_store.db_pool.runInteraction(
"_setup_events_stream_seqs", _setup_events_stream_seqs_set_pos,
)
synapse/__init__.py
+1 -1
View File
@@ -48,7 +48,7 @@ try:
except ImportError:
pass
__version__ = "1.22.1"
__version__ = "1.23.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when
synapse/config/saml2_config.py
+22
View File
@@ -271,6 +271,28 @@ class SAML2Config(Config):
#description: ["My awesome SP", "en"]
#name: ["Test SP", "en"]
#ui_info:
# display_name:
# - lang: en
# text: "Display Name is the descriptive name of your service."
# description:
# - lang: en
# text: "Description should be a short paragraph explaining the purpose of the service."
# information_url:
# - lang: en
# text: "https://example.com/terms-of-service"
# privacy_statement_url:
# - lang: en
# text: "https://example.com/privacy-policy"
# keywords:
# - lang: en
# text: ["Matrix", "Element"]
# logo:
# - lang: en
# text: "https://example.com/logo.svg"
# width: "200"
# height: "80"
#organization:
# name: Example com
# display_name:
synapse/metrics/__init__.py
+10
View File
@@ -502,6 +502,16 @@ build_info.labels(
last_ticked = time.time()
# 3PID send info
threepid_send_requests = Histogram(
"synapse_threepid_send_requests_with_tries",
documentation="Number of requests for a 3pid token by try count. Note if"
" there is a request with try count of 4, then there would have been one"
" each for 1, 2 and 3",
buckets=(1, 2, 3, 4, 5, 10),
labelnames=("type", "reason"),
)
class ReactorLastSeenMetric:
def collect(self):
synapse/rest/client/v2_alpha/account.py
+13
View File
@@ -39,6 +39,7 @@ from synapse.http.servlet import (
parse_json_object_from_request,
parse_string,
)
from synapse.metrics import threepid_send_requests
from synapse.push.mailer import Mailer
from synapse.types import UserID
from synapse.util.msisdn import phone_number_to_msisdn
@@ -145,6 +146,10 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
threepid_send_requests.labels(type="email", reason="password_reset").observe(
send_attempt
)
return 200, ret
@@ -441,6 +446,10 @@ class EmailThreepidRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
threepid_send_requests.labels(type="email", reason="add_threepid").observe(
send_attempt
)
return 200, ret
@@ -511,6 +520,10 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
next_link,
)
threepid_send_requests.labels(type="msisdn", reason="add_threepid").observe(
send_attempt
)
return 200, ret
synapse/rest/client/v2_alpha/register.py
+9
View File
@@ -47,6 +47,7 @@ from synapse.http.servlet import (
parse_json_object_from_request,
parse_string,
)
from synapse.metrics import threepid_send_requests
from synapse.push.mailer import Mailer
from synapse.util.msisdn import phone_number_to_msisdn
from synapse.util.ratelimitutils import FederationRateLimiter
@@ -165,6 +166,10 @@ class EmailRegisterRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
threepid_send_requests.labels(type="email", reason="register").observe(
send_attempt
)
return 200, ret
@@ -238,6 +243,10 @@ class MsisdnRegisterRequestTokenRestServlet(RestServlet):
next_link,
)
threepid_send_requests.labels(type="msisdn", reason="register").observe(
send_attempt
)
return 200, ret
synapse/storage/database.py
+6 -1
View File
@@ -88,13 +88,18 @@ def make_pool(
"""Get the connection pool for the database.
"""
# By default enable `cp_reconnect`. We need to fiddle with db_args in case
# someone has explicitly set `cp_reconnect`.
db_args = dict(db_config.config.get("args", {}))
db_args.setdefault("cp_reconnect", True)
return adbapi.ConnectionPool(
db_config.config["name"],
cp_reactor=reactor,
cp_openfun=lambda conn: engine.on_new_connection(
LoggingDatabaseConnection(conn, engine, "on_new_connection")
),
**db_config.config.get("args", {}),
**db_args,
)
synapse/storage/databases/main/event_federation.py
+70 -12
View File
@@ -26,6 +26,7 @@ from synapse.storage.databases.main.events_worker import EventsWorkerStore
from synapse.storage.databases.main.signatures import SignatureWorkerStore
from synapse.types import Collection
from synapse.util.caches.descriptors import cached
from synapse.util.caches.lrucache import LruCache
from synapse.util.iterutils import batch_iter
logger = logging.getLogger(__name__)
@@ -40,6 +41,11 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
self._delete_old_forward_extrem_cache, 60 * 60 * 1000
)
# Cache of event ID to list of auth event IDs and their depths.
self._event_auth_cache = LruCache(
500000, "_event_auth_cache", size_callback=len
) # type: LruCache[str, List[Tuple[str, int]]]
async def get_auth_chain(
self, event_ids: Collection[str], include_given: bool = False
) -> List[EventBase]:
@@ -84,17 +90,45 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
else:
results = set()
base_sql = "SELECT DISTINCT auth_id FROM event_auth WHERE "
# We pull out the depth simply so that we can populate the
# `_event_auth_cache` cache.
base_sql = """
SELECT a.event_id, auth_id, depth
FROM event_auth AS a
INNER JOIN events AS e ON (e.event_id = a.auth_id)
WHERE
"""
front = set(event_ids)
while front:
new_front = set()
for chunk in batch_iter(front, 100):
clause, args = make_in_list_sql_clause(
txn.database_engine, "event_id", chunk
)
txn.execute(base_sql + clause, args)
new_front.update(r[0] for r in txn)
# Pull the auth events either from the cache or DB.
to_fetch = [] # Event IDs to fetch from DB # type: List[str]
for event_id in chunk:
res = self._event_auth_cache.get(event_id)
if res is None:
to_fetch.append(event_id)
else:
new_front.update(auth_id for auth_id, depth in res)
if to_fetch:
clause, args = make_in_list_sql_clause(
txn.database_engine, "a.event_id", to_fetch
)
txn.execute(base_sql + clause, args)
# Note we need to batch up the results by event ID before
# adding to the cache.
to_cache = {}
for event_id, auth_event_id, auth_event_depth in txn:
to_cache.setdefault(event_id, []).append(
(auth_event_id, auth_event_depth)
)
new_front.add(auth_event_id)
for event_id, auth_events in to_cache.items():
self._event_auth_cache.set(event_id, auth_events)
new_front -= results
@@ -213,14 +247,38 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
break
# Fetch the auth events and their depths of the N last events we're
# currently walking
# currently walking, either from cache or DB.
search, chunk = search[:-100], search[-100:]
clause, args = make_in_list_sql_clause(
txn.database_engine, "a.event_id", [e_id for _, e_id in chunk]
)
txn.execute(base_sql + clause, args)
for event_id, auth_event_id, auth_event_depth in txn:
found = [] # Results found # type: List[Tuple[str, str, int]]
to_fetch = [] # Event IDs to fetch from DB # type: List[str]
for _, event_id in chunk:
res = self._event_auth_cache.get(event_id)
if res is None:
to_fetch.append(event_id)
else:
found.extend((event_id, auth_id, depth) for auth_id, depth in res)
if to_fetch:
clause, args = make_in_list_sql_clause(
txn.database_engine, "a.event_id", to_fetch
)
txn.execute(base_sql + clause, args)
# We parse the results and add the to the `found` set and the
# cache (note we need to batch up the results by event ID before
# adding to the cache).
to_cache = {}
for event_id, auth_event_id, auth_event_depth in txn:
to_cache.setdefault(event_id, []).append(
(auth_event_id, auth_event_depth)
)
found.append((event_id, auth_event_id, auth_event_depth))
for event_id, auth_events in to_cache.items():
self._event_auth_cache.set(event_id, auth_events)
for event_id, auth_event_id, auth_event_depth in found:
event_to_auth_events.setdefault(event_id, set()).add(auth_event_id)
sets = event_to_missing_sets.get(auth_event_id)