Merge remote-tracking branch 'origin/release-v1.20.0' into bbz/info-mainline-1.20.0
This commit is contained in:
Ben Banfield-Zanin
@@ -4,18 +4,16 @@ jobs:
|
||||
machine: true
|
||||
steps:
|
||||
- checkout
|
||||
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
|
||||
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} .
|
||||
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
|
||||
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
|
||||
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
|
||||
dockerhubuploadlatest:
|
||||
machine: true
|
||||
steps:
|
||||
- checkout
|
||||
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
|
||||
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest .
|
||||
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
|
||||
- run: docker push matrixdotorg/synapse:latest
|
||||
- run: docker push matrixdotorg/synapse:latest-py3
|
||||
|
||||
workflows:
|
||||
version: 2
|
||||
|
||||
4
.github/ISSUE_TEMPLATE/BUG_REPORT.md
vendored
4
.github/ISSUE_TEMPLATE/BUG_REPORT.md
vendored
@@ -4,12 +4,12 @@ about: Create a report to help us improve
|
||||
|
||||
---
|
||||
|
||||
<!--
|
||||
|
||||
**THIS IS NOT A SUPPORT CHANNEL!**
|
||||
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
|
||||
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)
|
||||
|
||||
<!--
|
||||
|
||||
If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/
|
||||
|
||||
This is a bug report template. By following the instructions below and
|
||||
|
||||
509
CHANGES.md
509
CHANGES.md
@@ -1,3 +1,512 @@
|
||||
Synapse 1.20.0rc3 (2020-09-11)
|
||||
==============================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
|
||||
|
||||
|
||||
Synapse 1.20.0rc2 (2020-09-09)
|
||||
==============================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. ([\#8280](https://github.com/matrix-org/synapse/issues/8280))
|
||||
|
||||
|
||||
Synapse 1.20.0rc1 (2020-09-08)
|
||||
==============================
|
||||
|
||||
Removal warning
|
||||
---------------
|
||||
|
||||
Some older clients used a [disallowed character](https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-register-email-requesttoken) (`:`) in the `client_secret` parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Add an endpoint to query your shared rooms with another user as an implementation of [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666). ([\#7785](https://github.com/matrix-org/synapse/issues/7785))
|
||||
- Iteratively encode JSON to avoid blocking the reactor. ([\#8013](https://github.com/matrix-org/synapse/issues/8013), [\#8116](https://github.com/matrix-org/synapse/issues/8116))
|
||||
- Add support for shadow-banning users (ignoring any message send requests). ([\#8034](https://github.com/matrix-org/synapse/issues/8034), [\#8092](https://github.com/matrix-org/synapse/issues/8092), [\#8095](https://github.com/matrix-org/synapse/issues/8095), [\#8142](https://github.com/matrix-org/synapse/issues/8142), [\#8152](https://github.com/matrix-org/synapse/issues/8152), [\#8157](https://github.com/matrix-org/synapse/issues/8157), [\#8158](https://github.com/matrix-org/synapse/issues/8158), [\#8176](https://github.com/matrix-org/synapse/issues/8176))
|
||||
- Use the default template file when its equivalent is not found in a custom template directory. ([\#8037](https://github.com/matrix-org/synapse/issues/8037), [\#8107](https://github.com/matrix-org/synapse/issues/8107), [\#8252](https://github.com/matrix-org/synapse/issues/8252))
|
||||
- Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654). ([\#8059](https://github.com/matrix-org/synapse/issues/8059), [\#8254](https://github.com/matrix-org/synapse/issues/8254), [\#8270](https://github.com/matrix-org/synapse/issues/8270), [\#8274](https://github.com/matrix-org/synapse/issues/8274))
|
||||
- Optimise `/federation/v1/user/devices/` API by only returning devices with encryption keys. ([\#8198](https://github.com/matrix-org/synapse/issues/8198))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable. ([\#7864](https://github.com/matrix-org/synapse/issues/7864))
|
||||
- Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed. ([\#8081](https://github.com/matrix-org/synapse/issues/8081))
|
||||
- Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints. ([\#8101](https://github.com/matrix-org/synapse/issues/8101))
|
||||
- Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. ([\#8104](https://github.com/matrix-org/synapse/issues/8104))
|
||||
- Fix a long-standing bug where invalid JSON would be accepted by Synapse. ([\#8106](https://github.com/matrix-org/synapse/issues/8106))
|
||||
- Fix a bug introduced in Synapse v1.12.0 which could cause `/sync` requests to fail with a 404 if you had a very old outstanding room invite. ([\#8110](https://github.com/matrix-org/synapse/issues/8110))
|
||||
- Return a proper error code when the rooms of an invalid group are requested. ([\#8129](https://github.com/matrix-org/synapse/issues/8129))
|
||||
- Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. ([\#8131](https://github.com/matrix-org/synapse/issues/8131))
|
||||
- Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. ([\#8135](https://github.com/matrix-org/synapse/issues/8135))
|
||||
- Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
|
||||
- Fix logging in via OpenID Connect with a provider that uses integer user IDs. ([\#8190](https://github.com/matrix-org/synapse/issues/8190))
|
||||
- Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. ([\#8223](https://github.com/matrix-org/synapse/issues/8223))
|
||||
- Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. ([\#8226](https://github.com/matrix-org/synapse/issues/8226))
|
||||
- Fix slow start times for large servers by removing a table scan of the `users` table from startup code. ([\#8271](https://github.com/matrix-org/synapse/issues/8271))
|
||||
|
||||
|
||||
Updates to the Docker image
|
||||
---------------------------
|
||||
|
||||
- Fix builds of the Docker image on non-x86 platforms. ([\#8144](https://github.com/matrix-org/synapse/issues/8144))
|
||||
- Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. ([\#8147](https://github.com/matrix-org/synapse/issues/8147))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Link to matrix-synapse-rest-password-provider in the password provider documentation. ([\#8111](https://github.com/matrix-org/synapse/issues/8111))
|
||||
- Updated documentation to note that Synapse does not follow `HTTP 308` redirects due to an upstream library not supporting them. Contributed by Ryan Cole. ([\#8120](https://github.com/matrix-org/synapse/issues/8120))
|
||||
- Explain better what GDPR-erased means when deactivating a user. ([\#8189](https://github.com/matrix-org/synapse/issues/8189))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7377](https://github.com/matrix-org/synapse/issues/7377), [\#8163](https://github.com/matrix-org/synapse/issues/8163))
|
||||
- Reduce run times of some unit tests by advancing the reactor a fewer number of times. ([\#7757](https://github.com/matrix-org/synapse/issues/7757))
|
||||
- Don't fail `/submit_token` requests on incorrect session ID if `request_token_inhibit_3pid_errors` is turned on. ([\#7991](https://github.com/matrix-org/synapse/issues/7991))
|
||||
- Convert various parts of the codebase to async/await. ([\#8071](https://github.com/matrix-org/synapse/issues/8071), [\#8072](https://github.com/matrix-org/synapse/issues/8072), [\#8074](https://github.com/matrix-org/synapse/issues/8074), [\#8075](https://github.com/matrix-org/synapse/issues/8075), [\#8076](https://github.com/matrix-org/synapse/issues/8076), [\#8087](https://github.com/matrix-org/synapse/issues/8087), [\#8100](https://github.com/matrix-org/synapse/issues/8100), [\#8119](https://github.com/matrix-org/synapse/issues/8119), [\#8121](https://github.com/matrix-org/synapse/issues/8121), [\#8133](https://github.com/matrix-org/synapse/issues/8133), [\#8156](https://github.com/matrix-org/synapse/issues/8156), [\#8162](https://github.com/matrix-org/synapse/issues/8162), [\#8166](https://github.com/matrix-org/synapse/issues/8166), [\#8168](https://github.com/matrix-org/synapse/issues/8168), [\#8173](https://github.com/matrix-org/synapse/issues/8173), [\#8191](https://github.com/matrix-org/synapse/issues/8191), [\#8192](https://github.com/matrix-org/synapse/issues/8192), [\#8193](https://github.com/matrix-org/synapse/issues/8193), [\#8194](https://github.com/matrix-org/synapse/issues/8194), [\#8195](https://github.com/matrix-org/synapse/issues/8195), [\#8197](https://github.com/matrix-org/synapse/issues/8197), [\#8199](https://github.com/matrix-org/synapse/issues/8199), [\#8200](https://github.com/matrix-org/synapse/issues/8200), [\#8201](https://github.com/matrix-org/synapse/issues/8201), [\#8202](https://github.com/matrix-org/synapse/issues/8202), [\#8207](https://github.com/matrix-org/synapse/issues/8207), [\#8213](https://github.com/matrix-org/synapse/issues/8213), [\#8214](https://github.com/matrix-org/synapse/issues/8214))
|
||||
- Remove some unused database functions. ([\#8085](https://github.com/matrix-org/synapse/issues/8085))
|
||||
- Add type hints to various parts of the codebase. ([\#8090](https://github.com/matrix-org/synapse/issues/8090), [\#8127](https://github.com/matrix-org/synapse/issues/8127), [\#8187](https://github.com/matrix-org/synapse/issues/8187), [\#8241](https://github.com/matrix-org/synapse/issues/8241), [\#8140](https://github.com/matrix-org/synapse/issues/8140), [\#8183](https://github.com/matrix-org/synapse/issues/8183), [\#8232](https://github.com/matrix-org/synapse/issues/8232), [\#8235](https://github.com/matrix-org/synapse/issues/8235), [\#8237](https://github.com/matrix-org/synapse/issues/8237), [\#8244](https://github.com/matrix-org/synapse/issues/8244))
|
||||
- Return the previous stream token if a non-member event is a duplicate. ([\#8093](https://github.com/matrix-org/synapse/issues/8093), [\#8112](https://github.com/matrix-org/synapse/issues/8112))
|
||||
- Separate `get_current_token` into two since there are two different use cases for it. ([\#8113](https://github.com/matrix-org/synapse/issues/8113))
|
||||
- Remove `ChainedIdGenerator`. ([\#8123](https://github.com/matrix-org/synapse/issues/8123))
|
||||
- Reduce the amount of whitespace in JSON stored and sent in responses. ([\#8124](https://github.com/matrix-org/synapse/issues/8124))
|
||||
- Update the test federation client to handle streaming responses. ([\#8130](https://github.com/matrix-org/synapse/issues/8130))
|
||||
- Micro-optimisations to `get_auth_chain_ids`. ([\#8132](https://github.com/matrix-org/synapse/issues/8132))
|
||||
- Refactor `StreamIdGenerator` and `MultiWriterIdGenerator` to have the same interface. ([\#8161](https://github.com/matrix-org/synapse/issues/8161))
|
||||
- Add functions to `MultiWriterIdGen` used by events stream. ([\#8164](https://github.com/matrix-org/synapse/issues/8164), [\#8179](https://github.com/matrix-org/synapse/issues/8179))
|
||||
- Fix tests that were broken due to the merge of 1.19.1. ([\#8167](https://github.com/matrix-org/synapse/issues/8167))
|
||||
- Make `SlavedIdTracker.advance` have the same interface as `MultiWriterIDGenerator`. ([\#8171](https://github.com/matrix-org/synapse/issues/8171))
|
||||
- Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`. ([\#8174](https://github.com/matrix-org/synapse/issues/8174), [\#8181](https://github.com/matrix-org/synapse/issues/8181))
|
||||
- Standardize the mypy configuration. ([\#8175](https://github.com/matrix-org/synapse/issues/8175))
|
||||
- Refactor some of `LoginRestServlet`'s helper methods, and move them to `AuthHandler` for easier reuse. ([\#8182](https://github.com/matrix-org/synapse/issues/8182))
|
||||
- Fix `wait_for_stream_position` to allow multiple waiters on same stream ID. ([\#8196](https://github.com/matrix-org/synapse/issues/8196))
|
||||
- Make `MultiWriterIDGenerator` work for streams that use negative values. ([\#8203](https://github.com/matrix-org/synapse/issues/8203))
|
||||
- Refactor queries for device keys and cross-signatures. ([\#8204](https://github.com/matrix-org/synapse/issues/8204), [\#8205](https://github.com/matrix-org/synapse/issues/8205), [\#8222](https://github.com/matrix-org/synapse/issues/8222), [\#8224](https://github.com/matrix-org/synapse/issues/8224), [\#8225](https://github.com/matrix-org/synapse/issues/8225), [\#8231](https://github.com/matrix-org/synapse/issues/8231), [\#8233](https://github.com/matrix-org/synapse/issues/8233), [\#8234](https://github.com/matrix-org/synapse/issues/8234))
|
||||
- Fix type hints for functions decorated with `@cached`. ([\#8240](https://github.com/matrix-org/synapse/issues/8240))
|
||||
- Remove obsolete `order` field from federation send queues. ([\#8245](https://github.com/matrix-org/synapse/issues/8245))
|
||||
- Stop sub-classing from object. ([\#8249](https://github.com/matrix-org/synapse/issues/8249))
|
||||
- Add more logging to debug slow startup. ([\#8264](https://github.com/matrix-org/synapse/issues/8264))
|
||||
- Do not attempt to upgrade database schema on worker processes. ([\#8266](https://github.com/matrix-org/synapse/issues/8266), [\#8276](https://github.com/matrix-org/synapse/issues/8276))
|
||||
|
||||
|
||||
Synapse 1.19.1 (2020-08-27)
|
||||
===========================
|
||||
|
||||
No significant changes.
|
||||
|
||||
|
||||
Synapse 1.19.1rc1 (2020-08-25)
|
||||
==============================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a bug introduced in v1.19.0 where appservices with ratelimiting disabled would still be ratelimited when joining rooms. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
|
||||
- Fix a bug introduced in v1.19.0 that would cause e.g. profile updates to fail due to incorrect application of rate limits on join requests. ([\#8153](https://github.com/matrix-org/synapse/issues/8153))
|
||||
|
||||
|
||||
Synapse 1.19.0 (2020-08-17)
|
||||
===========================
|
||||
|
||||
No significant changes since 1.19.0rc1.
|
||||
|
||||
Removal warning
|
||||
---------------
|
||||
|
||||
As outlined in the [previous release](https://github.com/matrix-org/synapse/releases/tag/v1.18.0), we are no longer publishing Docker images with the `-py3` tag suffix. On top of that, we have also removed the `latest-py3` tag. Please see [the announcement in the upgrade notes for 1.18.0](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180).
|
||||
|
||||
|
||||
Synapse 1.19.0rc1 (2020-08-13)
|
||||
==============================
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Add option to allow server admins to join rooms which fail complexity checks. Contributed by @lugino-emeritus. ([\#7902](https://github.com/matrix-org/synapse/issues/7902))
|
||||
- Add an option to purge room or not with delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel. ([\#7964](https://github.com/matrix-org/synapse/issues/7964))
|
||||
- Add rate limiting to users joining rooms. ([\#8008](https://github.com/matrix-org/synapse/issues/8008))
|
||||
- Add a `/health` endpoint to every configured HTTP listener that can be used as a health check endpoint by load balancers. ([\#8048](https://github.com/matrix-org/synapse/issues/8048))
|
||||
- Allow login to be blocked based on the values of SAML attributes. ([\#8052](https://github.com/matrix-org/synapse/issues/8052))
|
||||
- Allow guest access to the `GET /_matrix/client/r0/rooms/{room_id}/members` endpoint, according to MSC2689. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7314](https://github.com/matrix-org/synapse/issues/7314))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix a bug introduced in Synapse v1.7.2 which caused inaccurate membership counts in the room directory. ([\#7977](https://github.com/matrix-org/synapse/issues/7977))
|
||||
- Fix a long standing bug: 'Duplicate key value violates unique constraint "event_relations_id"' when message retention is configured. ([\#7978](https://github.com/matrix-org/synapse/issues/7978))
|
||||
- Fix "no create event in auth events" when trying to reject invitation after inviter leaves. Bug introduced in Synapse v1.10.0. ([\#7980](https://github.com/matrix-org/synapse/issues/7980))
|
||||
- Fix various comments and minor discrepencies in server notices code. ([\#7996](https://github.com/matrix-org/synapse/issues/7996))
|
||||
- Fix a long standing bug where HTTP HEAD requests resulted in a 400 error. ([\#7999](https://github.com/matrix-org/synapse/issues/7999))
|
||||
- Fix a long-standing bug which caused two copies of some log lines to be written when synctl was used along with a MemoryHandler logger. ([\#8011](https://github.com/matrix-org/synapse/issues/8011), [\#8012](https://github.com/matrix-org/synapse/issues/8012))
|
||||
|
||||
|
||||
Updates to the Docker image
|
||||
---------------------------
|
||||
|
||||
- We no longer publish Docker images with the `-py3` tag suffix, as [announced in the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180). ([\#8056](https://github.com/matrix-org/synapse/issues/8056))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Document how to set up a client .well-known file and fix several pieces of outdated documentation. ([\#7899](https://github.com/matrix-org/synapse/issues/7899))
|
||||
- Improve workers docs. ([\#7990](https://github.com/matrix-org/synapse/issues/7990), [\#8000](https://github.com/matrix-org/synapse/issues/8000))
|
||||
- Fix typo in `docs/workers.md`. ([\#7992](https://github.com/matrix-org/synapse/issues/7992))
|
||||
- Add documentation for how to undo a room shutdown. ([\#7998](https://github.com/matrix-org/synapse/issues/7998), [\#8010](https://github.com/matrix-org/synapse/issues/8010))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Reduce the amount of whitespace in JSON stored and sent in responses. Contributed by David Vo. ([\#7372](https://github.com/matrix-org/synapse/issues/7372))
|
||||
- Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0. ([\#7936](https://github.com/matrix-org/synapse/issues/7936), [\#7979](https://github.com/matrix-org/synapse/issues/7979))
|
||||
- Convert various parts of the codebase to async/await. ([\#7947](https://github.com/matrix-org/synapse/issues/7947), [\#7948](https://github.com/matrix-org/synapse/issues/7948), [\#7949](https://github.com/matrix-org/synapse/issues/7949), [\#7951](https://github.com/matrix-org/synapse/issues/7951), [\#7963](https://github.com/matrix-org/synapse/issues/7963), [\#7973](https://github.com/matrix-org/synapse/issues/7973), [\#7975](https://github.com/matrix-org/synapse/issues/7975), [\#7976](https://github.com/matrix-org/synapse/issues/7976), [\#7981](https://github.com/matrix-org/synapse/issues/7981), [\#7987](https://github.com/matrix-org/synapse/issues/7987), [\#7989](https://github.com/matrix-org/synapse/issues/7989), [\#8003](https://github.com/matrix-org/synapse/issues/8003), [\#8014](https://github.com/matrix-org/synapse/issues/8014), [\#8016](https://github.com/matrix-org/synapse/issues/8016), [\#8027](https://github.com/matrix-org/synapse/issues/8027), [\#8031](https://github.com/matrix-org/synapse/issues/8031), [\#8032](https://github.com/matrix-org/synapse/issues/8032), [\#8035](https://github.com/matrix-org/synapse/issues/8035), [\#8042](https://github.com/matrix-org/synapse/issues/8042), [\#8044](https://github.com/matrix-org/synapse/issues/8044), [\#8045](https://github.com/matrix-org/synapse/issues/8045), [\#8061](https://github.com/matrix-org/synapse/issues/8061), [\#8062](https://github.com/matrix-org/synapse/issues/8062), [\#8063](https://github.com/matrix-org/synapse/issues/8063), [\#8066](https://github.com/matrix-org/synapse/issues/8066), [\#8069](https://github.com/matrix-org/synapse/issues/8069), [\#8070](https://github.com/matrix-org/synapse/issues/8070))
|
||||
- Move some database-related log lines from the default logger to the database/transaction loggers. ([\#7952](https://github.com/matrix-org/synapse/issues/7952))
|
||||
- Add a script to detect source code files using non-unix line terminators. ([\#7965](https://github.com/matrix-org/synapse/issues/7965), [\#7970](https://github.com/matrix-org/synapse/issues/7970))
|
||||
- Log the SAML session ID during creation. ([\#7971](https://github.com/matrix-org/synapse/issues/7971))
|
||||
- Implement new experimental push rules for some users. ([\#7997](https://github.com/matrix-org/synapse/issues/7997))
|
||||
- Remove redundant and unreliable signature check for v1 Identity Service lookup responses. ([\#8001](https://github.com/matrix-org/synapse/issues/8001))
|
||||
- Improve the performance of the register endpoint. ([\#8009](https://github.com/matrix-org/synapse/issues/8009))
|
||||
- Reduce less useful output in the newsfragment CI step. Add a link to the changelog section of the contributing guide on error. ([\#8024](https://github.com/matrix-org/synapse/issues/8024))
|
||||
- Rename storage layer objects to be more sensible. ([\#8033](https://github.com/matrix-org/synapse/issues/8033))
|
||||
- Change the default log config to reduce disk I/O and storage for new servers. ([\#8040](https://github.com/matrix-org/synapse/issues/8040))
|
||||
- Add an assertion on `prev_events` in `create_new_client_event`. ([\#8041](https://github.com/matrix-org/synapse/issues/8041))
|
||||
- Add a comment to `ServerContextFactory` about the use of `SSLv23_METHOD`. ([\#8043](https://github.com/matrix-org/synapse/issues/8043))
|
||||
- Log `OPTIONS` requests at `DEBUG` rather than `INFO` level to reduce amount logged at `INFO`. ([\#8049](https://github.com/matrix-org/synapse/issues/8049))
|
||||
- Reduce amount of outbound request logging at `INFO` level. ([\#8050](https://github.com/matrix-org/synapse/issues/8050))
|
||||
- It is no longer necessary to explicitly define `filters` in the logging configuration. (Continuing to do so is redundant but harmless.) ([\#8051](https://github.com/matrix-org/synapse/issues/8051))
|
||||
- Add and improve type hints. ([\#8058](https://github.com/matrix-org/synapse/issues/8058), [\#8064](https://github.com/matrix-org/synapse/issues/8064), [\#8060](https://github.com/matrix-org/synapse/issues/8060), [\#8067](https://github.com/matrix-org/synapse/issues/8067))
|
||||
|
||||
|
||||
Synapse 1.18.0 (2020-07-30)
|
||||
===========================
|
||||
|
||||
Deprecation Warnings
|
||||
--------------------
|
||||
|
||||
### Docker Tags with `-py3` Suffix
|
||||
|
||||
From 10th August 2020, we will no longer publish Docker images with the `-py3` tag suffix. The images tagged with the `-py3` suffix have been identical to the non-suffixed tags since release 0.99.0, and the suffix is obsolete.
|
||||
|
||||
On 10th August, we will remove the `latest-py3` tag. Existing per-release tags (such as `v1.18.0-py3`) will not be removed, but no new `-py3` tags will be added.
|
||||
|
||||
Scripts relying on the `-py3` suffix will need to be updated.
|
||||
|
||||
|
||||
### TCP-based Replication
|
||||
|
||||
When setting up worker processes, we now recommend the use of a Redis server for replication. The old direct TCP connection method is deprecated and will be removed in a future release. See [docs/workers.md](https://github.com/matrix-org/synapse/blob/release-v1.18.0/docs/workers.md) for more details.
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Update worker docs with latest enhancements. ([\#7969](https://github.com/matrix-org/synapse/issues/7969))
|
||||
|
||||
|
||||
Synapse 1.18.0rc2 (2020-07-28)
|
||||
==============================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix an `AssertionError` exception introduced in v1.18.0rc1. ([\#7876](https://github.com/matrix-org/synapse/issues/7876))
|
||||
- Fix experimental support for moving typing off master when worker is restarted, which is broken in v1.18.0rc1. ([\#7967](https://github.com/matrix-org/synapse/issues/7967))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Further optimise queueing of inbound replication commands. ([\#7876](https://github.com/matrix-org/synapse/issues/7876))
|
||||
|
||||
|
||||
Synapse 1.18.0rc1 (2020-07-27)
|
||||
==============================
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Include room states on invite events that are sent to application services. Contributed by @Sorunome. ([\#6455](https://github.com/matrix-org/synapse/issues/6455))
|
||||
- Add delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel. ([\#7613](https://github.com/matrix-org/synapse/issues/7613), [\#7953](https://github.com/matrix-org/synapse/issues/7953))
|
||||
- Add experimental support for running multiple federation sender processes. ([\#7798](https://github.com/matrix-org/synapse/issues/7798))
|
||||
- Add the option to validate the `iss` and `aud` claims for JWT logins. ([\#7827](https://github.com/matrix-org/synapse/issues/7827))
|
||||
- Add support for handling registration requests across multiple client reader workers. ([\#7830](https://github.com/matrix-org/synapse/issues/7830))
|
||||
- Add an admin API to list the users in a room. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7842](https://github.com/matrix-org/synapse/issues/7842))
|
||||
- Allow email subjects to be customised through Synapse's configuration. ([\#7846](https://github.com/matrix-org/synapse/issues/7846))
|
||||
- Add the ability to re-activate an account from the admin API. ([\#7847](https://github.com/matrix-org/synapse/issues/7847), [\#7908](https://github.com/matrix-org/synapse/issues/7908))
|
||||
- Add experimental support for running multiple pusher workers. ([\#7855](https://github.com/matrix-org/synapse/issues/7855))
|
||||
- Add experimental support for moving typing off master. ([\#7869](https://github.com/matrix-org/synapse/issues/7869), [\#7959](https://github.com/matrix-org/synapse/issues/7959))
|
||||
- Report CPU metrics to prometheus for time spent processing replication commands. ([\#7879](https://github.com/matrix-org/synapse/issues/7879))
|
||||
- Support oEmbed for media previews. ([\#7920](https://github.com/matrix-org/synapse/issues/7920))
|
||||
- Abort federation requests where the client disconnects before the ratelimiter expires. ([\#7930](https://github.com/matrix-org/synapse/issues/7930))
|
||||
- Cache responses to `/_matrix/federation/v1/state_ids` to reduce duplicated work. ([\#7931](https://github.com/matrix-org/synapse/issues/7931))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix detection of out of sync remote device lists when receiving events from remote users. ([\#7815](https://github.com/matrix-org/synapse/issues/7815))
|
||||
- Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain. ([\#7817](https://github.com/matrix-org/synapse/issues/7817))
|
||||
- Fix a bug causing Synapse to misinterpret the value `off` for `encryption_enabled_by_default_for_room_type` in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0. ([\#7822](https://github.com/matrix-org/synapse/issues/7822))
|
||||
- Fix bug where we did not always pass in `app_name` or `server_name` to email templates, including e.g. for registration emails. ([\#7829](https://github.com/matrix-org/synapse/issues/7829))
|
||||
- Errors which occur while using the non-standard JWT login now return the proper error: `403 Forbidden` with an error code of `M_FORBIDDEN`. ([\#7844](https://github.com/matrix-org/synapse/issues/7844))
|
||||
- Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0. ([\#7850](https://github.com/matrix-org/synapse/issues/7850))
|
||||
- Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation. ([\#7854](https://github.com/matrix-org/synapse/issues/7854))
|
||||
- Fix a bug which allowed empty rooms to be rejoined over federation. ([\#7859](https://github.com/matrix-org/synapse/issues/7859))
|
||||
- Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers. ([\#7866](https://github.com/matrix-org/synapse/issues/7866))
|
||||
- Fix a long standing bug where the tracing of async functions with opentracing was broken. ([\#7872](https://github.com/matrix-org/synapse/issues/7872), [\#7961](https://github.com/matrix-org/synapse/issues/7961))
|
||||
- Fix "TypeError in `synapse.notifier`" exceptions. ([\#7880](https://github.com/matrix-org/synapse/issues/7880))
|
||||
- Fix deprecation warning due to invalid escape sequences. ([\#7895](https://github.com/matrix-org/synapse/issues/7895))
|
||||
|
||||
|
||||
Updates to the Docker image
|
||||
---------------------------
|
||||
|
||||
- Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196. ([\#7839](https://github.com/matrix-org/synapse/issues/7839))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Provide instructions on using `register_new_matrix_user` via docker. ([\#7885](https://github.com/matrix-org/synapse/issues/7885))
|
||||
- Change the sample config postgres user section to use `synapse_user` instead of `synapse` to align with the documentation. ([\#7889](https://github.com/matrix-org/synapse/issues/7889))
|
||||
- Reorder database paragraphs to promote postgres over sqlite. ([\#7933](https://github.com/matrix-org/synapse/issues/7933))
|
||||
- Update the dates of ACME v1's end of life in [`ACME.md`](https://github.com/matrix-org/synapse/blob/master/docs/ACME.md). ([\#7934](https://github.com/matrix-org/synapse/issues/7934))
|
||||
|
||||
|
||||
Deprecations and Removals
|
||||
-------------------------
|
||||
|
||||
- Remove unused `synapse_replication_tcp_resource_invalidate_cache` prometheus metric. ([\#7878](https://github.com/matrix-org/synapse/issues/7878))
|
||||
- Remove Ubuntu Eoan from the list of `.deb` packages that we build as it is now end-of-life. Contributed by @gary-kim. ([\#7888](https://github.com/matrix-org/synapse/issues/7888))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Switch parts of the codebase from `simplejson` to the standard library `json`. ([\#7802](https://github.com/matrix-org/synapse/issues/7802))
|
||||
- Add type hints to the http server code and remove an unused parameter. ([\#7813](https://github.com/matrix-org/synapse/issues/7813))
|
||||
- Add type hints to synapse.api.errors module. ([\#7820](https://github.com/matrix-org/synapse/issues/7820))
|
||||
- Ensure that calls to `json.dumps` are compatible with the standard library json. ([\#7836](https://github.com/matrix-org/synapse/issues/7836))
|
||||
- Remove redundant `retry_on_integrity_error` wrapper for event persistence code. ([\#7848](https://github.com/matrix-org/synapse/issues/7848))
|
||||
- Consistently use `db_to_json` to convert from database values to JSON objects. ([\#7849](https://github.com/matrix-org/synapse/issues/7849))
|
||||
- Convert various parts of the codebase to async/await. ([\#7851](https://github.com/matrix-org/synapse/issues/7851), [\#7860](https://github.com/matrix-org/synapse/issues/7860), [\#7868](https://github.com/matrix-org/synapse/issues/7868), [\#7871](https://github.com/matrix-org/synapse/issues/7871), [\#7873](https://github.com/matrix-org/synapse/issues/7873), [\#7874](https://github.com/matrix-org/synapse/issues/7874), [\#7884](https://github.com/matrix-org/synapse/issues/7884), [\#7912](https://github.com/matrix-org/synapse/issues/7912), [\#7935](https://github.com/matrix-org/synapse/issues/7935), [\#7939](https://github.com/matrix-org/synapse/issues/7939), [\#7942](https://github.com/matrix-org/synapse/issues/7942), [\#7944](https://github.com/matrix-org/synapse/issues/7944))
|
||||
- Add support for handling registration requests across multiple client reader workers. ([\#7853](https://github.com/matrix-org/synapse/issues/7853))
|
||||
- Small performance improvement in typing processing. ([\#7856](https://github.com/matrix-org/synapse/issues/7856))
|
||||
- The default value of `filter_timeline_limit` was changed from -1 (no limit) to 100. ([\#7858](https://github.com/matrix-org/synapse/issues/7858))
|
||||
- Optimise queueing of inbound replication commands. ([\#7861](https://github.com/matrix-org/synapse/issues/7861))
|
||||
- Add some type annotations to `HomeServer` and `BaseHandler`. ([\#7870](https://github.com/matrix-org/synapse/issues/7870))
|
||||
- Clean up `PreserveLoggingContext`. ([\#7877](https://github.com/matrix-org/synapse/issues/7877))
|
||||
- Change "unknown room version" logging from 'error' to 'warning'. ([\#7881](https://github.com/matrix-org/synapse/issues/7881))
|
||||
- Stop using `device_max_stream_id` table and just use `device_inbox.stream_id`. ([\#7882](https://github.com/matrix-org/synapse/issues/7882))
|
||||
- Return an empty body for OPTIONS requests. ([\#7886](https://github.com/matrix-org/synapse/issues/7886))
|
||||
- Fix typo in generated config file. Contributed by @ThiefMaster. ([\#7890](https://github.com/matrix-org/synapse/issues/7890))
|
||||
- Import ABC from `collections.abc` for Python 3.10 compatibility. ([\#7892](https://github.com/matrix-org/synapse/issues/7892))
|
||||
- Remove unused functions `time_function`, `trace_function`, `get_previous_frames`
|
||||
and `get_previous_frame` from `synapse.logging.utils` module. ([\#7897](https://github.com/matrix-org/synapse/issues/7897))
|
||||
- Lint the `contrib/` directory in CI and linting scripts, add `synctl` to the linting script for consistency with CI. ([\#7914](https://github.com/matrix-org/synapse/issues/7914))
|
||||
- Use Element CSS and logo in notification emails when app name is Element. ([\#7919](https://github.com/matrix-org/synapse/issues/7919))
|
||||
- Optimisation to /sync handling: skip serializing the response if the client has already disconnected. ([\#7927](https://github.com/matrix-org/synapse/issues/7927))
|
||||
- When a client disconnects, don't log it as 'Error processing request'. ([\#7928](https://github.com/matrix-org/synapse/issues/7928))
|
||||
- Add debugging to `/sync` response generation (disabled by default). ([\#7929](https://github.com/matrix-org/synapse/issues/7929))
|
||||
- Update comments that refer to Deferreds for async functions. ([\#7945](https://github.com/matrix-org/synapse/issues/7945))
|
||||
- Simplify error handling in federation handler. ([\#7950](https://github.com/matrix-org/synapse/issues/7950))
|
||||
|
||||
|
||||
Synapse 1.17.0 (2020-07-13)
|
||||
===========================
|
||||
|
||||
Synapse 1.17.0 is identical to 1.17.0rc1, with the addition of the fix that was included in 1.16.1.
|
||||
|
||||
|
||||
Synapse 1.16.1 (2020-07-10)
|
||||
===========================
|
||||
|
||||
In some distributions of Synapse 1.16.0, we incorrectly included a database migration which added a new, unused table. This release removes the redundant table.
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Drop table `local_rejections_stream` which was incorrectly added in Synapse 1.16.0. ([\#7816](https://github.com/matrix-org/synapse/issues/7816), [b1beb3ff5](https://github.com/matrix-org/synapse/commit/b1beb3ff5))
|
||||
|
||||
|
||||
Synapse 1.17.0rc1 (2020-07-09)
|
||||
==============================
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel. ([\#7021](https://github.com/matrix-org/synapse/issues/7021))
|
||||
- Fix "Tried to close a non-active scope!" error messages when opentracing is enabled. ([\#7732](https://github.com/matrix-org/synapse/issues/7732))
|
||||
- Fix incorrect error message when database CTYPE was set incorrectly. ([\#7760](https://github.com/matrix-org/synapse/issues/7760))
|
||||
- Fix to not ignore `set_tweak` actions in Push Rules that have no `value`, as permitted by the specification. ([\#7766](https://github.com/matrix-org/synapse/issues/7766))
|
||||
- Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian. ([\#7779](https://github.com/matrix-org/synapse/issues/7779))
|
||||
- Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent. ([\#7797](https://github.com/matrix-org/synapse/issues/7797))
|
||||
- Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation. ([\#7804](https://github.com/matrix-org/synapse/issues/7804), [\#7809](https://github.com/matrix-org/synapse/issues/7809), [\#7810](https://github.com/matrix-org/synapse/issues/7810))
|
||||
|
||||
|
||||
Updates to the Docker image
|
||||
---------------------------
|
||||
|
||||
- Include libwebp in the Docker file to properly handle webp image uploads. ([\#7791](https://github.com/matrix-org/synapse/issues/7791))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Improve the documentation of the non-standard JSON web token login type. ([\#7776](https://github.com/matrix-org/synapse/issues/7776))
|
||||
- Update doc links for caddy. Contributed by Nicolai Søborg. ([\#7789](https://github.com/matrix-org/synapse/issues/7789))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Refactor getting replication updates from database. ([\#7740](https://github.com/matrix-org/synapse/issues/7740))
|
||||
- Send push notifications with a high or low priority depending upon whether they may generate user-observable effects. ([\#7765](https://github.com/matrix-org/synapse/issues/7765))
|
||||
- Use symbolic names for replication stream names. ([\#7768](https://github.com/matrix-org/synapse/issues/7768))
|
||||
- Add early returns to `_check_for_soft_fail`. ([\#7769](https://github.com/matrix-org/synapse/issues/7769))
|
||||
- Fix up `synapse.handlers.federation` to pass mypy. ([\#7770](https://github.com/matrix-org/synapse/issues/7770))
|
||||
- Convert the appserver handler to async/await. ([\#7775](https://github.com/matrix-org/synapse/issues/7775))
|
||||
- Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz. ([\#7780](https://github.com/matrix-org/synapse/issues/7780))
|
||||
- Update linting scripts and codebase to be compatible with `isort` v5. ([\#7786](https://github.com/matrix-org/synapse/issues/7786))
|
||||
- Stop populating unused table `local_invites`. ([\#7793](https://github.com/matrix-org/synapse/issues/7793))
|
||||
- Ensure that strings (not bytes) are passed into JSON serialization. ([\#7799](https://github.com/matrix-org/synapse/issues/7799))
|
||||
- Switch from simplejson to the standard library json. ([\#7800](https://github.com/matrix-org/synapse/issues/7800))
|
||||
- Add `signing_key` property to `HomeServer` to save code duplication. ([\#7805](https://github.com/matrix-org/synapse/issues/7805))
|
||||
- Improve stacktraces from exceptions in background processes. ([\#7808](https://github.com/matrix-org/synapse/issues/7808))
|
||||
- Fix various spelling errors in comments and log lines. ([\#7811](https://github.com/matrix-org/synapse/issues/7811))
|
||||
|
||||
|
||||
Synapse 1.16.0 (2020-07-08)
|
||||
===========================
|
||||
|
||||
No significant changes since 1.16.0rc2.
|
||||
|
||||
Note that this release deprecates the `m.login.jwt` login method, renaming it
|
||||
to `org.matrix.login.jwt`, as `m.login.jwt` is not part of the Matrix spec.
|
||||
Otherwise the behaviour is identical. Synapse will accept both names for now,
|
||||
but this may change in a future release.
|
||||
|
||||
Synapse 1.16.0rc2 (2020-07-02)
|
||||
==============================
|
||||
|
||||
Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
|
||||
Please see [below](#synapse-1152-2020-07-02) for more details.
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
|
||||
|
||||
|
||||
Synapse 1.15.2 (2020-07-02)
|
||||
===========================
|
||||
|
||||
Due to the two security issues highlighted below, server administrators are
|
||||
encouraged to update Synapse. We are not aware of these vulnerabilities being
|
||||
exploited in the wild.
|
||||
|
||||
Security advisory
|
||||
-----------------
|
||||
|
||||
* A malicious homeserver could force Synapse to reset the state in a room to a
|
||||
small subset of the correct state. This affects all Synapse deployments which
|
||||
federate with untrusted servers. ([96e9afe6](https://github.com/matrix-org/synapse/commit/96e9afe62500310977dc3cbc99a8d16d3d2fa15c))
|
||||
* HTML pages served via Synapse were vulnerable to clickjacking attacks. This
|
||||
predominantly affects homeservers with single-sign-on enabled, but all server
|
||||
administrators are encouraged to upgrade. ([ea26e9a9](https://github.com/matrix-org/synapse/commit/ea26e9a98b0541fc886a1cb826a38352b7599dbe))
|
||||
|
||||
This was reported by [Quentin Gliech](https://sandhose.fr/).
|
||||
|
||||
|
||||
Synapse 1.16.0rc1 (2020-07-01)
|
||||
==============================
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Add an option to enable encryption by default for new rooms. ([\#7639](https://github.com/matrix-org/synapse/issues/7639))
|
||||
- Add support for running multiple media repository workers. See [docs/workers.md](https://github.com/matrix-org/synapse/blob/release-v1.16.0/docs/workers.md) for instructions. ([\#7706](https://github.com/matrix-org/synapse/issues/7706))
|
||||
- Media can now be marked as safe from quarantined. ([\#7718](https://github.com/matrix-org/synapse/issues/7718))
|
||||
- Expand the configuration options for auto-join rooms. ([\#7763](https://github.com/matrix-org/synapse/issues/7763))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Remove `user_id` from the response to `GET /_matrix/client/r0/presence/{userId}/status` to match the specification. ([\#7606](https://github.com/matrix-org/synapse/issues/7606))
|
||||
- In worker mode, ensure that replicated data has not already been received. ([\#7648](https://github.com/matrix-org/synapse/issues/7648))
|
||||
- Fix intermittent exception during startup, introduced in Synapse 1.14.0. ([\#7663](https://github.com/matrix-org/synapse/issues/7663))
|
||||
- Include a user-agent for federation and well-known requests. ([\#7677](https://github.com/matrix-org/synapse/issues/7677))
|
||||
- Accept the proper field (`phone`) for the `m.id.phone` identifier type. The legacy field of `number` is still accepted as a fallback. Bug introduced in v0.20.0. ([\#7687](https://github.com/matrix-org/synapse/issues/7687))
|
||||
- Fix "Starting db txn 'get_completed_ui_auth_stages' from sentinel context" warning. The bug was introduced in 1.13.0. ([\#7688](https://github.com/matrix-org/synapse/issues/7688))
|
||||
- Compare the URI and method during user interactive authentication (instead of the URI twice). Bug introduced in 1.13.0. ([\#7689](https://github.com/matrix-org/synapse/issues/7689))
|
||||
- Fix a long standing bug where the response to the `GET room_keys/version` endpoint had the incorrect type for the `etag` field. ([\#7691](https://github.com/matrix-org/synapse/issues/7691))
|
||||
- Fix logged error during device resync in opentracing. Broke in v1.14.0. ([\#7698](https://github.com/matrix-org/synapse/issues/7698))
|
||||
- Do not break push rule evaluation when receiving an event with a non-string body. This is a long-standing bug. ([\#7701](https://github.com/matrix-org/synapse/issues/7701))
|
||||
- Fixs a long standing bug which resulted in an exception: "TypeError: argument of type 'ObservableDeferred' is not iterable". ([\#7708](https://github.com/matrix-org/synapse/issues/7708))
|
||||
- The `synapse_port_db` script no longer fails when the `ui_auth_sessions` table is non-empty. This bug has existed since v1.13.0. ([\#7711](https://github.com/matrix-org/synapse/issues/7711))
|
||||
- Synapse will now fetch media from the proper specified URL (using the r0 prefix instead of the unspecified v1). ([\#7714](https://github.com/matrix-org/synapse/issues/7714))
|
||||
- Fix the tables ignored by `synapse_port_db` to be in sync the current database schema. ([\#7717](https://github.com/matrix-org/synapse/issues/7717))
|
||||
- Fix missing `Content-Length` on HTTP responses from the metrics handler. ([\#7730](https://github.com/matrix-org/synapse/issues/7730))
|
||||
- Fix large state resolutions from stalling Synapse for seconds at a time. ([\#7735](https://github.com/matrix-org/synapse/issues/7735), [\#7746](https://github.com/matrix-org/synapse/issues/7746))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Spelling correction in sample_config.yaml. ([\#7652](https://github.com/matrix-org/synapse/issues/7652))
|
||||
- Added instructions for how to use Keycloak via OpenID Connect to authenticate with Synapse. ([\#7659](https://github.com/matrix-org/synapse/issues/7659))
|
||||
- Corrected misspelling of PostgreSQL. ([\#7724](https://github.com/matrix-org/synapse/issues/7724))
|
||||
|
||||
|
||||
Deprecations and Removals
|
||||
-------------------------
|
||||
|
||||
- Deprecate `m.login.jwt` login method in favour of `org.matrix.login.jwt`, as `m.login.jwt` is not part of the Matrix spec. ([\#7675](https://github.com/matrix-org/synapse/issues/7675))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Refactor getting replication updates from database. ([\#7636](https://github.com/matrix-org/synapse/issues/7636))
|
||||
- Clean-up the login fallback code. ([\#7657](https://github.com/matrix-org/synapse/issues/7657))
|
||||
- Increase the default SAML session expiry time to 15 minutes. ([\#7664](https://github.com/matrix-org/synapse/issues/7664))
|
||||
- Convert the device message and pagination handlers to async/await. ([\#7678](https://github.com/matrix-org/synapse/issues/7678))
|
||||
- Convert typing handler to async/await. ([\#7679](https://github.com/matrix-org/synapse/issues/7679))
|
||||
- Require `parameterized` package version to be at least 0.7.0. ([\#7680](https://github.com/matrix-org/synapse/issues/7680))
|
||||
- Refactor handling of `listeners` configuration settings. ([\#7681](https://github.com/matrix-org/synapse/issues/7681))
|
||||
- Replace uses of `six.iterkeys`/`iteritems`/`itervalues` with `keys()`/`items()`/`values()`. ([\#7692](https://github.com/matrix-org/synapse/issues/7692))
|
||||
- Add support for using `rust-python-jaeger-reporter` library to reduce jaeger tracing overhead. ([\#7697](https://github.com/matrix-org/synapse/issues/7697))
|
||||
- Make Tox actions work on Debian 10. ([\#7703](https://github.com/matrix-org/synapse/issues/7703))
|
||||
- Replace all remaining uses of `six` with native Python 3 equivalents. Contributed by @ilmari. ([\#7704](https://github.com/matrix-org/synapse/issues/7704))
|
||||
- Fix broken link in sample config. ([\#7712](https://github.com/matrix-org/synapse/issues/7712))
|
||||
- Speed up state res v2 across large state differences. ([\#7725](https://github.com/matrix-org/synapse/issues/7725))
|
||||
- Convert directory handler to async/await. ([\#7727](https://github.com/matrix-org/synapse/issues/7727))
|
||||
- Move `flake8` to the end of `scripts-dev/lint.sh` as it takes the longest and could cause the script to exit early. ([\#7738](https://github.com/matrix-org/synapse/issues/7738))
|
||||
- Explain the "test" conditional requirement for dependencies is not all of the modules necessary to run the unit tests. ([\#7751](https://github.com/matrix-org/synapse/issues/7751))
|
||||
- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7755](https://github.com/matrix-org/synapse/issues/7755))
|
||||
|
||||
|
||||
Synapse 1.15.1 (2020-06-16)
|
||||
===========================
|
||||
|
||||
|
||||
121
INSTALL.md
121
INSTALL.md
@@ -1,10 +1,12 @@
|
||||
- [Choosing your server name](#choosing-your-server-name)
|
||||
- [Picking a database engine](#picking-a-database-engine)
|
||||
- [Installing Synapse](#installing-synapse)
|
||||
- [Installing from source](#installing-from-source)
|
||||
- [Platform-Specific Instructions](#platform-specific-instructions)
|
||||
- [Prebuilt packages](#prebuilt-packages)
|
||||
- [Setting up Synapse](#setting-up-synapse)
|
||||
- [TLS certificates](#tls-certificates)
|
||||
- [Client Well-Known URI](#client-well-known-uri)
|
||||
- [Email](#email)
|
||||
- [Registering a user](#registering-a-user)
|
||||
- [Setting up a TURN server](#setting-up-a-turn-server)
|
||||
@@ -27,6 +29,25 @@ that your email address is probably `user@example.com` rather than
|
||||
`user@email.example.com`) - but doing so may require more advanced setup: see
|
||||
[Setting up Federation](docs/federate.md).
|
||||
|
||||
# Picking a database engine
|
||||
|
||||
Synapse offers two database engines:
|
||||
* [PostgreSQL](https://www.postgresql.org)
|
||||
* [SQLite](https://sqlite.org/)
|
||||
|
||||
Almost all installations should opt to use PostgreSQL. Advantages include:
|
||||
|
||||
* significant performance improvements due to the superior threading and
|
||||
caching model, smarter query optimiser
|
||||
* allowing the DB to be run on separate hardware
|
||||
|
||||
For information on how to install and use PostgreSQL, please see
|
||||
[docs/postgres.md](docs/postgres.md)
|
||||
|
||||
By default Synapse uses SQLite and in doing so trades performance for convenience.
|
||||
SQLite is only recommended in Synapse for testing purposes or for servers with
|
||||
light workloads.
|
||||
|
||||
# Installing Synapse
|
||||
|
||||
## Installing from source
|
||||
@@ -234,9 +255,9 @@ for a number of platforms.
|
||||
|
||||
There is an offical synapse image available at
|
||||
https://hub.docker.com/r/matrixdotorg/synapse which can be used with
|
||||
the docker-compose file available at [contrib/docker](contrib/docker). Further information on
|
||||
this including configuration options is available in the README on
|
||||
hub.docker.com.
|
||||
the docker-compose file available at [contrib/docker](contrib/docker). Further
|
||||
information on this including configuration options is available in the README
|
||||
on hub.docker.com.
|
||||
|
||||
Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
|
||||
Dockerfile to automate a synapse server in a single Docker image, at
|
||||
@@ -244,7 +265,8 @@ https://hub.docker.com/r/avhost/docker-matrix/tags/
|
||||
|
||||
Slavi Pantaleev has created an Ansible playbook,
|
||||
which installs the offical Docker image of Matrix Synapse
|
||||
along with many other Matrix-related services (Postgres database, riot-web, coturn, mxisd, SSL support, etc.).
|
||||
along with many other Matrix-related services (Postgres database, Element, coturn,
|
||||
ma1sd, SSL support, etc.).
|
||||
For more details, see
|
||||
https://github.com/spantaleev/matrix-docker-ansible-deploy
|
||||
|
||||
@@ -277,22 +299,27 @@ The fingerprint of the repository signing key (as shown by `gpg
|
||||
/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
|
||||
`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
|
||||
|
||||
#### Downstream Debian/Ubuntu packages
|
||||
#### Downstream Debian packages
|
||||
|
||||
For `buster` and `sid`, Synapse is available in the Debian repositories and
|
||||
it should be possible to install it with simply:
|
||||
We do not recommend using the packages from the default Debian `buster`
|
||||
repository at this time, as they are old and suffer from known security
|
||||
vulnerabilities. You can install the latest version of Synapse from
|
||||
[our repository](#matrixorg-packages) or from `buster-backports`. Please
|
||||
see the [Debian documentation](https://backports.debian.org/Instructions/)
|
||||
for information on how to use backports.
|
||||
|
||||
If you are using Debian `sid` or testing, Synapse is available in the default
|
||||
repositories and it should be possible to install it simply with:
|
||||
|
||||
```
|
||||
sudo apt install matrix-synapse
|
||||
```
|
||||
|
||||
There is also a version of `matrix-synapse` in `stretch-backports`. Please see
|
||||
the [Debian documentation on
|
||||
backports](https://backports.debian.org/Instructions/) for information on how
|
||||
to use them.
|
||||
#### Downstream Ubuntu packages
|
||||
|
||||
We do not recommend using the packages in downstream Ubuntu at this time, as
|
||||
they are old and suffer from known security vulnerabilities.
|
||||
We do not recommend using the packages in the default Ubuntu repository
|
||||
at this time, as they are old and suffer from known security vulnerabilities.
|
||||
The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
|
||||
|
||||
### Fedora
|
||||
|
||||
@@ -405,13 +432,11 @@ so, you will need to edit `homeserver.yaml`, as follows:
|
||||
```
|
||||
|
||||
* You will also need to uncomment the `tls_certificate_path` and
|
||||
`tls_private_key_path` lines under the `TLS` section. You can either
|
||||
point these settings at an existing certificate and key, or you can
|
||||
enable Synapse's built-in ACME (Let's Encrypt) support. Instructions
|
||||
for having Synapse automatically provision and renew federation
|
||||
certificates through ACME can be found at [ACME.md](docs/ACME.md).
|
||||
Note that, as pointed out in that document, this feature will not
|
||||
work with installs set up after November 2019.
|
||||
`tls_private_key_path` lines under the `TLS` section. You will need to manage
|
||||
provisioning of these certificates yourself — Synapse had built-in ACME
|
||||
support, but the ACMEv1 protocol Synapse implements is deprecated, not
|
||||
allowed by LetsEncrypt for new sites, and will break for existing sites in
|
||||
late 2020. See [ACME.md](docs/ACME.md).
|
||||
|
||||
If you are using your own certificate, be sure to use a `.pem` file that
|
||||
includes the full certificate chain including any intermediate certificates
|
||||
@@ -421,6 +446,60 @@ so, you will need to edit `homeserver.yaml`, as follows:
|
||||
For a more detailed guide to configuring your server for federation, see
|
||||
[federate.md](docs/federate.md).
|
||||
|
||||
## Client Well-Known URI
|
||||
|
||||
Setting up the client Well-Known URI is optional but if you set it up, it will
|
||||
allow users to enter their full username (e.g. `@user:<server_name>`) into clients
|
||||
which support well-known lookup to automatically configure the homeserver and
|
||||
identity server URLs. This is useful so that users don't have to memorize or think
|
||||
about the actual homeserver URL you are using.
|
||||
|
||||
The URL `https://<server_name>/.well-known/matrix/client` should return JSON in
|
||||
the following format.
|
||||
|
||||
```
|
||||
{
|
||||
"m.homeserver": {
|
||||
"base_url": "https://<matrix.example.com>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
It can optionally contain identity server information as well.
|
||||
|
||||
```
|
||||
{
|
||||
"m.homeserver": {
|
||||
"base_url": "https://<matrix.example.com>"
|
||||
},
|
||||
"m.identity_server": {
|
||||
"base_url": "https://<identity.example.com>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
To work in browser based clients, the file must be served with the appropriate
|
||||
Cross-Origin Resource Sharing (CORS) headers. A recommended value would be
|
||||
`Access-Control-Allow-Origin: *` which would allow all browser based clients to
|
||||
view it.
|
||||
|
||||
In nginx this would be something like:
|
||||
```
|
||||
location /.well-known/matrix/client {
|
||||
return 200 '{"m.homeserver": {"base_url": "https://<matrix.example.com>"}}';
|
||||
add_header Content-Type application/json;
|
||||
add_header Access-Control-Allow-Origin *;
|
||||
}
|
||||
```
|
||||
|
||||
You should also ensure the `public_baseurl` option in `homeserver.yaml` is set
|
||||
correctly. `public_baseurl` should be set to the URL that clients will use to
|
||||
connect to your server. This is the same URL you put for the `m.homeserver`
|
||||
`base_url` above.
|
||||
|
||||
```
|
||||
public_baseurl: "https://<matrix.example.com>"
|
||||
```
|
||||
|
||||
## Email
|
||||
|
||||
@@ -439,7 +518,7 @@ email will be disabled.
|
||||
|
||||
## Registering a user
|
||||
|
||||
The easiest way to create a new user is to do so from a client like [Riot](https://riot.im).
|
||||
The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
|
||||
|
||||
Alternatively you can do so from the command line if you have installed via pip.
|
||||
|
||||
|
||||
45
README.rst
45
README.rst
@@ -45,7 +45,7 @@ which handle:
|
||||
- Eventually-consistent cryptographically secure synchronisation of room
|
||||
state across a global open network of federated servers and services
|
||||
- Sending and receiving extensible messages in a room with (optional)
|
||||
end-to-end encryption[1]
|
||||
end-to-end encryption
|
||||
- Inviting, joining, leaving, kicking, banning room members
|
||||
- Managing user accounts (registration, login, logout)
|
||||
- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
|
||||
@@ -82,9 +82,6 @@ at the `Matrix spec <https://matrix.org/docs/spec>`_, and experiment with the
|
||||
|
||||
Thanks for using Matrix!
|
||||
|
||||
[1] End-to-end encryption is currently in beta: `blog post <https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last>`_.
|
||||
|
||||
|
||||
Support
|
||||
=======
|
||||
|
||||
@@ -115,12 +112,11 @@ Unless you are running a test instance of Synapse on your local machine, in
|
||||
general, you will need to enable TLS support before you can successfully
|
||||
connect from a client: see `<INSTALL.md#tls-certificates>`_.
|
||||
|
||||
An easy way to get started is to login or register via Riot at
|
||||
https://riot.im/app/#/login or https://riot.im/app/#/register respectively.
|
||||
An easy way to get started is to login or register via Element at
|
||||
https://app.element.io/#/login or https://app.element.io/#/register respectively.
|
||||
You will need to change the server you are logging into from ``matrix.org``
|
||||
and instead specify a Homeserver URL of ``https://<server_name>:8448``
|
||||
(or just ``https://<server_name>`` if you are using a reverse proxy).
|
||||
(Leave the identity server as the default - see `Identity servers`_.)
|
||||
If you prefer to use another client, refer to our
|
||||
`client breakdown <https://matrix.org/docs/projects/clients-matrix>`_.
|
||||
|
||||
@@ -137,7 +133,7 @@ it, specify ``enable_registration: true`` in ``homeserver.yaml``. (It is then
|
||||
recommended to also set up CAPTCHA - see `<docs/CAPTCHA_SETUP.md>`_.)
|
||||
|
||||
Once ``enable_registration`` is set to ``true``, it is possible to register a
|
||||
user via `riot.im <https://riot.im/app/#/register>`_ or other Matrix clients.
|
||||
user via a Matrix client.
|
||||
|
||||
Your new user name will be formed partly from the ``server_name``, and partly
|
||||
from a localpart you specify when you create the account. Your name will take
|
||||
@@ -183,30 +179,6 @@ versions of synapse.
|
||||
|
||||
.. _UPGRADE.rst: UPGRADE.rst
|
||||
|
||||
|
||||
Using PostgreSQL
|
||||
================
|
||||
|
||||
Synapse offers two database engines:
|
||||
* `SQLite <https://sqlite.org/>`_
|
||||
* `PostgreSQL <https://www.postgresql.org>`_
|
||||
|
||||
By default Synapse uses SQLite in and doing so trades performance for convenience.
|
||||
SQLite is only recommended in Synapse for testing purposes or for servers with
|
||||
light workloads.
|
||||
|
||||
Almost all installations should opt to use PostreSQL. Advantages include:
|
||||
|
||||
* significant performance improvements due to the superior threading and
|
||||
caching model, smarter query optimiser
|
||||
* allowing the DB to be run on separate hardware
|
||||
* allowing basic active/backup high-availability with a "hot spare" synapse
|
||||
pointing at the same DB master, as well as enabling DB replication in
|
||||
synapse itself.
|
||||
|
||||
For information on how to install and use PostgreSQL, please see
|
||||
`docs/postgres.md <docs/postgres.md>`_.
|
||||
|
||||
.. _reverse-proxy:
|
||||
|
||||
Using a reverse proxy with Synapse
|
||||
@@ -215,7 +187,7 @@ Using a reverse proxy with Synapse
|
||||
It is recommended to put a reverse proxy such as
|
||||
`nginx <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>`_,
|
||||
`Apache <https://httpd.apache.org/docs/current/mod/mod_proxy_http.html>`_,
|
||||
`Caddy <https://caddyserver.com/docs/proxy>`_ or
|
||||
`Caddy <https://caddyserver.com/docs/quick-starts/reverse-proxy>`_ or
|
||||
`HAProxy <https://www.haproxy.org/>`_ in front of Synapse. One advantage of
|
||||
doing so is that it means that you can expose the default https port (443) to
|
||||
Matrix clients without needing to run Synapse with root privileges.
|
||||
@@ -255,10 +227,9 @@ email address.
|
||||
Password reset
|
||||
==============
|
||||
|
||||
If a user has registered an email address to their account using an identity
|
||||
server, they can request a password-reset token via clients such as Riot.
|
||||
|
||||
A manual password reset can be done via direct database access as follows.
|
||||
Users can reset their password through their client. Alternatively, a server admin
|
||||
can reset a users password using the `admin API <docs/admin_api/user_admin_api.rst#reset-password>`_
|
||||
or by directly editing the database as shown below.
|
||||
|
||||
First calculate the hash of the new password::
|
||||
|
||||
|
||||
18
UPGRADE.rst
18
UPGRADE.rst
@@ -75,6 +75,24 @@ for example:
|
||||
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
|
||||
Upgrading to v1.18.0
|
||||
====================
|
||||
|
||||
Docker `-py3` suffix will be removed in future versions
|
||||
-------------------------------------------------------
|
||||
|
||||
From 10th August 2020, we will no longer publish Docker images with the `-py3` tag suffix. The images tagged with the `-py3` suffix have been identical to the non-suffixed tags since release 0.99.0, and the suffix is obsolete.
|
||||
|
||||
On 10th August, we will remove the `latest-py3` tag. Existing per-release tags (such as `v1.18.0-py3`) will not be removed, but no new `-py3` tags will be added.
|
||||
|
||||
Scripts relying on the `-py3` suffix will need to be updated.
|
||||
|
||||
Redis replication is now recommended in lieu of TCP replication
|
||||
---------------------------------------------------------------
|
||||
|
||||
When setting up worker processes, we now recommend the use of a Redis server for replication. **The old direct TCP connection method is deprecated and will be removed in a future release.**
|
||||
See `docs/workers.md <docs/workers.md>`_ for more details.
|
||||
|
||||
Upgrading to v1.14.0
|
||||
====================
|
||||
|
||||
|
||||
@@ -17,9 +17,6 @@
|
||||
""" Starts a synapse client console. """
|
||||
from __future__ import print_function
|
||||
|
||||
from twisted.internet import reactor, defer, threads
|
||||
from http import TwistedHttpClient
|
||||
|
||||
import argparse
|
||||
import cmd
|
||||
import getpass
|
||||
@@ -28,12 +25,14 @@ import shlex
|
||||
import sys
|
||||
import time
|
||||
import urllib
|
||||
import urlparse
|
||||
from http import TwistedHttpClient
|
||||
|
||||
import nacl.signing
|
||||
import nacl.encoding
|
||||
import nacl.signing
|
||||
import urlparse
|
||||
from signedjson.sign import SignatureVerifyException, verify_signed_json
|
||||
|
||||
from signedjson.sign import verify_signed_json, SignatureVerifyException
|
||||
from twisted.internet import defer, reactor, threads
|
||||
|
||||
CONFIG_JSON = "cmdclient_config.json"
|
||||
|
||||
@@ -493,7 +492,7 @@ class SynapseCmd(cmd.Cmd):
|
||||
"list messages <roomid> from=END&to=START&limit=3"
|
||||
"""
|
||||
args = self._parse(line, ["type", "roomid", "qp"])
|
||||
if not "type" in args or not "roomid" in args:
|
||||
if "type" not in args or "roomid" not in args:
|
||||
print("Must specify type and room ID.")
|
||||
return
|
||||
if args["type"] not in ["members", "messages"]:
|
||||
@@ -508,7 +507,7 @@ class SynapseCmd(cmd.Cmd):
|
||||
try:
|
||||
key_value = key_value_str.split("=")
|
||||
qp[key_value[0]] = key_value[1]
|
||||
except:
|
||||
except Exception:
|
||||
print("Bad query param: %s" % key_value)
|
||||
return
|
||||
|
||||
@@ -585,7 +584,7 @@ class SynapseCmd(cmd.Cmd):
|
||||
parsed_url = urlparse.urlparse(args["path"])
|
||||
qp.update(urlparse.parse_qs(parsed_url.query))
|
||||
args["path"] = parsed_url.path
|
||||
except:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
reactor.callFromThread(
|
||||
@@ -610,13 +609,15 @@ class SynapseCmd(cmd.Cmd):
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _do_event_stream(self, timeout):
|
||||
res = yield self.http_client.get_json(
|
||||
self._url() + "/events",
|
||||
{
|
||||
"access_token": self._tok(),
|
||||
"timeout": str(timeout),
|
||||
"from": self.event_stream_token,
|
||||
},
|
||||
res = yield defer.ensureDeferred(
|
||||
self.http_client.get_json(
|
||||
self._url() + "/events",
|
||||
{
|
||||
"access_token": self._tok(),
|
||||
"timeout": str(timeout),
|
||||
"from": self.event_stream_token,
|
||||
},
|
||||
)
|
||||
)
|
||||
print(json.dumps(res, indent=4))
|
||||
|
||||
@@ -772,10 +773,10 @@ def main(server_url, identity_server_url, username, token, config_path):
|
||||
syn_cmd.config = json.load(config)
|
||||
try:
|
||||
http_client.verbose = "on" == syn_cmd.config["verbose"]
|
||||
except:
|
||||
except Exception:
|
||||
pass
|
||||
print("Loaded config from %s" % config_path)
|
||||
except:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Twisted-specific: Runs the command processor in Twisted's event loop
|
||||
|
||||
@@ -14,17 +14,17 @@
|
||||
# limitations under the License.
|
||||
|
||||
from __future__ import print_function
|
||||
from twisted.web.client import Agent, readBody
|
||||
from twisted.web.http_headers import Headers
|
||||
from twisted.internet import defer, reactor
|
||||
|
||||
from pprint import pformat
|
||||
|
||||
import json
|
||||
import urllib
|
||||
from pprint import pformat
|
||||
|
||||
from twisted.internet import defer, reactor
|
||||
from twisted.web.client import Agent, readBody
|
||||
from twisted.web.http_headers import Headers
|
||||
|
||||
|
||||
class HttpClient(object):
|
||||
class HttpClient:
|
||||
""" Interface for talking json over http
|
||||
"""
|
||||
|
||||
@@ -169,7 +169,7 @@ class TwistedHttpClient(HttpClient):
|
||||
return d
|
||||
|
||||
|
||||
class _RawProducer(object):
|
||||
class _RawProducer:
|
||||
def __init__(self, data):
|
||||
self.data = data
|
||||
self.body = data
|
||||
@@ -186,7 +186,7 @@ class _RawProducer(object):
|
||||
pass
|
||||
|
||||
|
||||
class _JsonProducer(object):
|
||||
class _JsonProducer:
|
||||
""" Used by the twisted http client to create the HTTP body from json
|
||||
"""
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ services:
|
||||
- traefik.http.routers.https-synapse.tls.certResolver=le-ssl
|
||||
|
||||
db:
|
||||
image: docker.io/postgres:10-alpine
|
||||
image: docker.io/postgres:12-alpine
|
||||
# Change that password, of course!
|
||||
environment:
|
||||
- POSTGRES_USER=synapse
|
||||
|
||||
@@ -141,7 +141,7 @@ class CursesStdIO:
|
||||
curses.endwin()
|
||||
|
||||
|
||||
class Callback(object):
|
||||
class Callback:
|
||||
def __init__(self, stdio):
|
||||
self.stdio = stdio
|
||||
|
||||
|
||||
@@ -28,27 +28,24 @@ Currently assumes the local address is localhost:<port>
|
||||
"""
|
||||
|
||||
|
||||
from synapse.federation import ReplicationHandler
|
||||
|
||||
from synapse.federation.units import Pdu
|
||||
|
||||
from synapse.util import origin_from_ucid
|
||||
|
||||
from synapse.app.homeserver import SynapseHomeServer
|
||||
|
||||
# from synapse.logging.utils import log_function
|
||||
|
||||
from twisted.internet import reactor, defer
|
||||
from twisted.python import log
|
||||
|
||||
import argparse
|
||||
import curses.wrapper
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
import cursesio
|
||||
import curses.wrapper
|
||||
|
||||
from twisted.internet import defer, reactor
|
||||
from twisted.python import log
|
||||
|
||||
from synapse.app.homeserver import SynapseHomeServer
|
||||
from synapse.federation import ReplicationHandler
|
||||
from synapse.federation.units import Pdu
|
||||
from synapse.util import origin_from_ucid
|
||||
|
||||
# from synapse.logging.utils import log_function
|
||||
|
||||
|
||||
logger = logging.getLogger("example")
|
||||
@@ -58,7 +55,7 @@ def excpetion_errback(failure):
|
||||
logging.exception(failure)
|
||||
|
||||
|
||||
class InputOutput(object):
|
||||
class InputOutput:
|
||||
""" This is responsible for basic I/O so that a user can interact with
|
||||
the example app.
|
||||
"""
|
||||
@@ -75,7 +72,7 @@ class InputOutput(object):
|
||||
"""
|
||||
|
||||
try:
|
||||
m = re.match("^join (\S+)$", line)
|
||||
m = re.match(r"^join (\S+)$", line)
|
||||
if m:
|
||||
# The `sender` wants to join a room.
|
||||
(room_name,) = m.groups()
|
||||
@@ -84,7 +81,7 @@ class InputOutput(object):
|
||||
# self.print_line("OK.")
|
||||
return
|
||||
|
||||
m = re.match("^invite (\S+) (\S+)$", line)
|
||||
m = re.match(r"^invite (\S+) (\S+)$", line)
|
||||
if m:
|
||||
# `sender` wants to invite someone to a room
|
||||
room_name, invitee = m.groups()
|
||||
@@ -93,7 +90,7 @@ class InputOutput(object):
|
||||
# self.print_line("OK.")
|
||||
return
|
||||
|
||||
m = re.match("^send (\S+) (.*)$", line)
|
||||
m = re.match(r"^send (\S+) (.*)$", line)
|
||||
if m:
|
||||
# `sender` wants to message a room
|
||||
room_name, body = m.groups()
|
||||
@@ -102,7 +99,7 @@ class InputOutput(object):
|
||||
# self.print_line("OK.")
|
||||
return
|
||||
|
||||
m = re.match("^backfill (\S+)$", line)
|
||||
m = re.match(r"^backfill (\S+)$", line)
|
||||
if m:
|
||||
# we want to backfill a room
|
||||
(room_name,) = m.groups()
|
||||
@@ -135,7 +132,7 @@ class IOLoggerHandler(logging.Handler):
|
||||
self.io.print_log(msg)
|
||||
|
||||
|
||||
class Room(object):
|
||||
class Room:
|
||||
""" Used to store (in memory) the current membership state of a room, and
|
||||
which home servers we should send PDUs associated with the room to.
|
||||
"""
|
||||
@@ -201,16 +198,6 @@ class HomeServer(ReplicationHandler):
|
||||
% (pdu.context, pdu.pdu_type, json.dumps(pdu.content))
|
||||
)
|
||||
|
||||
# def on_state_change(self, pdu):
|
||||
##self.output.print_line("#%s (state) %s *** %s" %
|
||||
##(pdu.context, pdu.state_key, pdu.pdu_type)
|
||||
##)
|
||||
|
||||
# if "joinee" in pdu.content:
|
||||
# self._on_join(pdu.context, pdu.content["joinee"])
|
||||
# elif "invitee" in pdu.content:
|
||||
# self._on_invite(pdu.origin, pdu.context, pdu.content["invitee"])
|
||||
|
||||
def _on_message(self, pdu):
|
||||
""" We received a message
|
||||
"""
|
||||
@@ -314,7 +301,7 @@ class HomeServer(ReplicationHandler):
|
||||
return self.replication_layer.backfill(dest, room_name, limit)
|
||||
|
||||
def _get_room_remote_servers(self, room_name):
|
||||
return [i for i in self.joined_rooms.setdefault(room_name).servers]
|
||||
return list(self.joined_rooms.setdefault(room_name).servers)
|
||||
|
||||
def _get_or_create_room(self, room_name):
|
||||
return self.joined_rooms.setdefault(room_name, Room(room_name))
|
||||
@@ -334,7 +321,7 @@ def main(stdscr):
|
||||
user = args.user
|
||||
server_name = origin_from_ucid(user)
|
||||
|
||||
## Set up logging ##
|
||||
# Set up logging
|
||||
|
||||
root_logger = logging.getLogger()
|
||||
|
||||
@@ -354,7 +341,7 @@ def main(stdscr):
|
||||
observer = log.PythonLoggingObserver()
|
||||
observer.start()
|
||||
|
||||
## Set up synapse server
|
||||
# Set up synapse server
|
||||
|
||||
curses_stdio = cursesio.CursesStdIO(stdscr)
|
||||
input_output = InputOutput(curses_stdio, user)
|
||||
@@ -368,16 +355,16 @@ def main(stdscr):
|
||||
|
||||
input_output.set_home_server(hs)
|
||||
|
||||
## Add input_output logger
|
||||
# Add input_output logger
|
||||
io_logger = IOLoggerHandler(input_output)
|
||||
io_logger.setFormatter(formatter)
|
||||
root_logger.addHandler(io_logger)
|
||||
|
||||
## Start! ##
|
||||
# Start!
|
||||
|
||||
try:
|
||||
port = int(server_name.split(":")[1])
|
||||
except:
|
||||
except Exception:
|
||||
port = 12345
|
||||
|
||||
app_hs.get_http_server().start_listening(port)
|
||||
|
||||
@@ -1,7 +1,44 @@
|
||||
{
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_PROMETHEUS",
|
||||
"label": "Prometheus",
|
||||
"description": "",
|
||||
"type": "datasource",
|
||||
"pluginId": "prometheus",
|
||||
"pluginName": "Prometheus"
|
||||
}
|
||||
],
|
||||
"__requires": [
|
||||
{
|
||||
"type": "grafana",
|
||||
"id": "grafana",
|
||||
"name": "Grafana",
|
||||
"version": "6.7.4"
|
||||
},
|
||||
{
|
||||
"type": "panel",
|
||||
"id": "graph",
|
||||
"name": "Graph",
|
||||
"version": ""
|
||||
},
|
||||
{
|
||||
"type": "panel",
|
||||
"id": "heatmap",
|
||||
"name": "Heatmap",
|
||||
"version": ""
|
||||
},
|
||||
{
|
||||
"type": "datasource",
|
||||
"id": "prometheus",
|
||||
"name": "Prometheus",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"$$hashKey": "object:76",
|
||||
"builtIn": 1,
|
||||
"datasource": "$datasource",
|
||||
"enable": false,
|
||||
@@ -17,8 +54,8 @@
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 0,
|
||||
"id": 1,
|
||||
"iteration": 1591098104645,
|
||||
"id": null,
|
||||
"iteration": 1594646317221,
|
||||
"links": [
|
||||
{
|
||||
"asDropdown": true,
|
||||
@@ -34,7 +71,7 @@
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -269,7 +306,6 @@
|
||||
"show": false
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
@@ -559,7 +595,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -1423,7 +1459,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -1795,7 +1831,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -2531,7 +2567,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -2823,7 +2859,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -2844,7 +2880,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 33
|
||||
"y": 6
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 79,
|
||||
@@ -2940,7 +2976,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 33
|
||||
"y": 6
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 83,
|
||||
@@ -3038,7 +3074,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 42
|
||||
"y": 15
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 109,
|
||||
@@ -3137,7 +3173,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 42
|
||||
"y": 15
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 111,
|
||||
@@ -3223,14 +3259,14 @@
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "$datasource",
|
||||
"description": "",
|
||||
"description": "Number of events queued up on the master process for processing by the federation sender",
|
||||
"fill": 1,
|
||||
"fillGradient": 0,
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 51
|
||||
"y": 24
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 140,
|
||||
@@ -3354,6 +3390,103 @@
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"aliasColors": {},
|
||||
"bars": false,
|
||||
"dashLength": 10,
|
||||
"dashes": false,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"description": "The number of events in the in-memory queues ",
|
||||
"fill": 1,
|
||||
"fillGradient": 0,
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 24
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 142,
|
||||
"legend": {
|
||||
"avg": false,
|
||||
"current": false,
|
||||
"max": false,
|
||||
"min": false,
|
||||
"show": true,
|
||||
"total": false,
|
||||
"values": false
|
||||
},
|
||||
"lines": true,
|
||||
"linewidth": 1,
|
||||
"nullPointMode": "null",
|
||||
"options": {
|
||||
"dataLinks": []
|
||||
},
|
||||
"percentage": false,
|
||||
"pointradius": 2,
|
||||
"points": false,
|
||||
"renderer": "flot",
|
||||
"seriesOverrides": [],
|
||||
"spaceLength": 10,
|
||||
"stack": false,
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "synapse_federation_transaction_queue_pending_pdus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "pending PDUs {{job}}-{{index}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "synapse_federation_transaction_queue_pending_edus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "pending EDUs {{job}}-{{index}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"thresholds": [],
|
||||
"timeFrom": null,
|
||||
"timeRegions": [],
|
||||
"timeShift": null,
|
||||
"title": "In-memory federation transmission queues",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
"value_type": "individual"
|
||||
},
|
||||
"type": "graph",
|
||||
"xaxis": {
|
||||
"buckets": null,
|
||||
"mode": "time",
|
||||
"name": null,
|
||||
"show": true,
|
||||
"values": []
|
||||
},
|
||||
"yaxes": [
|
||||
{
|
||||
"$$hashKey": "object:317",
|
||||
"format": "short",
|
||||
"label": "events",
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": "0",
|
||||
"show": true
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:318",
|
||||
"format": "short",
|
||||
"label": "",
|
||||
"logBase": 1,
|
||||
"max": null,
|
||||
"min": null,
|
||||
"show": true
|
||||
}
|
||||
],
|
||||
"yaxis": {
|
||||
"align": false,
|
||||
"alignLevel": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"title": "Federation",
|
||||
@@ -3361,7 +3494,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -3567,7 +3700,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -3588,7 +3721,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 52
|
||||
"y": 79
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 48,
|
||||
@@ -3682,7 +3815,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 52
|
||||
"y": 79
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 104,
|
||||
@@ -3802,7 +3935,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 59
|
||||
"y": 86
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 10,
|
||||
@@ -3898,7 +4031,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 59
|
||||
"y": 86
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 11,
|
||||
@@ -3987,7 +4120,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -4011,7 +4144,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 67
|
||||
"y": 80
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 12,
|
||||
@@ -4106,7 +4239,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 67
|
||||
"y": 80
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 26,
|
||||
@@ -4201,7 +4334,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 80
|
||||
"y": 93
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 13,
|
||||
@@ -4297,7 +4430,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 80
|
||||
"y": 93
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 27,
|
||||
@@ -4392,7 +4525,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 93
|
||||
"y": 106
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 28,
|
||||
@@ -4486,7 +4619,7 @@
|
||||
"h": 13,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 93
|
||||
"y": 106
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 25,
|
||||
@@ -4572,7 +4705,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -5062,7 +5195,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -5083,7 +5216,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 66
|
||||
"y": 121
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 91,
|
||||
@@ -5179,7 +5312,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 66
|
||||
"y": 121
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 21,
|
||||
@@ -5271,7 +5404,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 75
|
||||
"y": 130
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 89,
|
||||
@@ -5369,7 +5502,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 75
|
||||
"y": 130
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 93,
|
||||
@@ -5459,7 +5592,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 84
|
||||
"y": 139
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 95,
|
||||
@@ -5552,12 +5685,12 @@
|
||||
"mode": "spectrum"
|
||||
},
|
||||
"dataFormat": "tsbuckets",
|
||||
"datasource": "Prometheus",
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 84
|
||||
"y": 139
|
||||
},
|
||||
"heatmap": {},
|
||||
"hideZeroBuckets": true,
|
||||
@@ -5567,7 +5700,6 @@
|
||||
"show": true
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
@@ -5609,7 +5741,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -5630,7 +5762,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 39
|
||||
"y": 66
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 2,
|
||||
@@ -5754,7 +5886,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 39
|
||||
"y": 66
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 41,
|
||||
@@ -5847,7 +5979,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 46
|
||||
"y": 73
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 42,
|
||||
@@ -5939,7 +6071,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 46
|
||||
"y": 73
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 43,
|
||||
@@ -6031,7 +6163,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 53
|
||||
"y": 80
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 113,
|
||||
@@ -6129,7 +6261,7 @@
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 53
|
||||
"y": 80
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 115,
|
||||
@@ -6215,7 +6347,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -6236,7 +6368,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 58
|
||||
"y": 40
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 67,
|
||||
@@ -6267,7 +6399,7 @@
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": " synapse_event_persisted_position{instance=\"$instance\",job=\"synapse\"} - ignoring(index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
|
||||
"expr": "max(synapse_event_persisted_position{instance=\"$instance\"}) - ignoring(instance,index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
|
||||
"format": "time_series",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
@@ -6328,7 +6460,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 58
|
||||
"y": 40
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 71,
|
||||
@@ -6362,6 +6494,7 @@
|
||||
"expr": "time()*1000-synapse_event_processing_last_ts{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
|
||||
"format": "time_series",
|
||||
"hide": false,
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "{{job}}-{{index}} {{name}}",
|
||||
"refId": "B"
|
||||
@@ -6420,7 +6553,7 @@
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 67
|
||||
"y": 49
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 121,
|
||||
@@ -6509,7 +6642,7 @@
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"datasource": null,
|
||||
"datasource": "${DS_PROMETHEUS}",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
@@ -6539,7 +6672,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 41
|
||||
"y": 86
|
||||
},
|
||||
"heatmap": {},
|
||||
"hideZeroBuckets": true,
|
||||
@@ -6549,7 +6682,6 @@
|
||||
"show": true
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
@@ -6599,7 +6731,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 41
|
||||
"y": 86
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 124,
|
||||
@@ -6700,7 +6832,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 49
|
||||
"y": 94
|
||||
},
|
||||
"heatmap": {},
|
||||
"hideZeroBuckets": true,
|
||||
@@ -6710,7 +6842,6 @@
|
||||
"show": true
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
@@ -6760,7 +6891,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 49
|
||||
"y": 94
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 128,
|
||||
@@ -6879,7 +7010,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 57
|
||||
"y": 102
|
||||
},
|
||||
"heatmap": {},
|
||||
"hideZeroBuckets": true,
|
||||
@@ -6889,7 +7020,6 @@
|
||||
"show": true
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
@@ -6939,7 +7069,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 57
|
||||
"y": 102
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 130,
|
||||
@@ -7058,7 +7188,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 65
|
||||
"y": 110
|
||||
},
|
||||
"heatmap": {},
|
||||
"hideZeroBuckets": true,
|
||||
@@ -7068,12 +7198,12 @@
|
||||
"show": true
|
||||
},
|
||||
"links": [],
|
||||
"options": {},
|
||||
"reverseYBuckets": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)",
|
||||
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size])",
|
||||
"format": "heatmap",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "{{le}}",
|
||||
"refId": "A"
|
||||
@@ -7118,7 +7248,7 @@
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 65
|
||||
"y": 110
|
||||
},
|
||||
"hiddenSeries": false,
|
||||
"id": 132,
|
||||
@@ -7149,29 +7279,33 @@
|
||||
"steppedLine": false,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)) ",
|
||||
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
|
||||
"format": "time_series",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "50%",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
|
||||
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
|
||||
"format": "time_series",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "75%",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
|
||||
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
|
||||
"format": "time_series",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "90%",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
|
||||
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
|
||||
"format": "time_series",
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "99%",
|
||||
"refId": "D"
|
||||
@@ -7181,7 +7315,7 @@
|
||||
"timeFrom": null,
|
||||
"timeRegions": [],
|
||||
"timeShift": null,
|
||||
"title": "Number of state resolution performed, by number of state groups involved (quantiles)",
|
||||
"title": "Number of state resolutions performed, by number of state groups involved (quantiles)",
|
||||
"tooltip": {
|
||||
"shared": true,
|
||||
"sort": 0,
|
||||
@@ -7233,6 +7367,7 @@
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
@@ -7309,14 +7444,12 @@
|
||||
},
|
||||
{
|
||||
"allValue": null,
|
||||
"current": {
|
||||
"text": "matrix.org",
|
||||
"value": "matrix.org"
|
||||
},
|
||||
"current": {},
|
||||
"datasource": "$datasource",
|
||||
"definition": "",
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"index": -1,
|
||||
"label": null,
|
||||
"multi": false,
|
||||
"name": "instance",
|
||||
@@ -7335,17 +7468,13 @@
|
||||
{
|
||||
"allFormat": "regex wildcard",
|
||||
"allValue": "",
|
||||
"current": {
|
||||
"text": "synapse",
|
||||
"value": [
|
||||
"synapse"
|
||||
]
|
||||
},
|
||||
"current": {},
|
||||
"datasource": "$datasource",
|
||||
"definition": "",
|
||||
"hide": 0,
|
||||
"hideLabel": false,
|
||||
"includeAll": true,
|
||||
"index": -1,
|
||||
"label": "Job",
|
||||
"multi": true,
|
||||
"multiFormat": "regex values",
|
||||
@@ -7366,16 +7495,13 @@
|
||||
{
|
||||
"allFormat": "regex wildcard",
|
||||
"allValue": ".*",
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
"current": {},
|
||||
"datasource": "$datasource",
|
||||
"definition": "",
|
||||
"hide": 0,
|
||||
"hideLabel": false,
|
||||
"includeAll": true,
|
||||
"index": -1,
|
||||
"label": "",
|
||||
"multi": true,
|
||||
"multiFormat": "regex values",
|
||||
@@ -7428,5 +7554,8 @@
|
||||
"timezone": "",
|
||||
"title": "Synapse",
|
||||
"uid": "000000012",
|
||||
"version": 29
|
||||
"variables": {
|
||||
"list": []
|
||||
},
|
||||
"version": 32
|
||||
}
|
||||
@@ -1,5 +1,13 @@
|
||||
from __future__ import print_function
|
||||
|
||||
import argparse
|
||||
import cgi
|
||||
import datetime
|
||||
import json
|
||||
|
||||
import pydot
|
||||
import urllib2
|
||||
|
||||
# Copyright 2014-2016 OpenMarket Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
@@ -15,15 +23,6 @@ from __future__ import print_function
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
import sqlite3
|
||||
import pydot
|
||||
import cgi
|
||||
import json
|
||||
import datetime
|
||||
import argparse
|
||||
import urllib2
|
||||
|
||||
|
||||
def make_name(pdu_id, origin):
|
||||
return "%s@%s" % (pdu_id, origin)
|
||||
|
||||
@@ -33,7 +32,7 @@ def make_graph(pdus, room, filename_prefix):
|
||||
node_map = {}
|
||||
|
||||
origins = set()
|
||||
colors = set(("red", "green", "blue", "yellow", "purple"))
|
||||
colors = {"red", "green", "blue", "yellow", "purple"}
|
||||
|
||||
for pdu in pdus:
|
||||
origins.add(pdu.get("origin"))
|
||||
@@ -49,7 +48,7 @@ def make_graph(pdus, room, filename_prefix):
|
||||
try:
|
||||
c = colors.pop()
|
||||
color_map[o] = c
|
||||
except:
|
||||
except Exception:
|
||||
print("Run out of colours!")
|
||||
color_map[o] = "black"
|
||||
|
||||
|
||||
@@ -13,12 +13,13 @@
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
import sqlite3
|
||||
import pydot
|
||||
import cgi
|
||||
import json
|
||||
import datetime
|
||||
import argparse
|
||||
import cgi
|
||||
import datetime
|
||||
import json
|
||||
import sqlite3
|
||||
|
||||
import pydot
|
||||
|
||||
from synapse.events import FrozenEvent
|
||||
from synapse.util.frozenutils import unfreeze
|
||||
@@ -98,7 +99,7 @@ def make_graph(db_name, room_id, file_prefix, limit):
|
||||
for prev_id, _ in event.prev_events:
|
||||
try:
|
||||
end_node = node_map[prev_id]
|
||||
except:
|
||||
except Exception:
|
||||
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
|
||||
|
||||
node_map[prev_id] = end_node
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
from __future__ import print_function
|
||||
|
||||
import argparse
|
||||
import cgi
|
||||
import datetime
|
||||
|
||||
import pydot
|
||||
import simplejson as json
|
||||
|
||||
from synapse.events import FrozenEvent
|
||||
from synapse.util.frozenutils import unfreeze
|
||||
|
||||
# Copyright 2016 OpenMarket Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
@@ -15,18 +25,6 @@ from __future__ import print_function
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
import pydot
|
||||
import cgi
|
||||
import simplejson as json
|
||||
import datetime
|
||||
import argparse
|
||||
|
||||
from synapse.events import FrozenEvent
|
||||
from synapse.util.frozenutils import unfreeze
|
||||
|
||||
from six import string_types
|
||||
|
||||
|
||||
def make_graph(file_name, room_id, file_prefix, limit):
|
||||
print("Reading lines")
|
||||
with open(file_name) as f:
|
||||
@@ -62,7 +60,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
|
||||
for key, value in unfreeze(event.get_dict()["content"]).items():
|
||||
if value is None:
|
||||
value = "<null>"
|
||||
elif isinstance(value, string_types):
|
||||
elif isinstance(value, str):
|
||||
pass
|
||||
else:
|
||||
value = json.dumps(value)
|
||||
@@ -108,7 +106,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
|
||||
for prev_id, _ in event.prev_events:
|
||||
try:
|
||||
end_node = node_map[prev_id]
|
||||
except:
|
||||
except Exception:
|
||||
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
|
||||
|
||||
node_map[prev_id] = end_node
|
||||
|
||||
@@ -12,15 +12,15 @@ npm install jquery jsdom
|
||||
"""
|
||||
from __future__ import print_function
|
||||
|
||||
import gevent
|
||||
import grequests
|
||||
from BeautifulSoup import BeautifulSoup
|
||||
import json
|
||||
import urllib
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
# ACCESS_TOKEN="" #
|
||||
import gevent
|
||||
import grequests
|
||||
from BeautifulSoup import BeautifulSoup
|
||||
|
||||
ACCESS_TOKEN = ""
|
||||
|
||||
MATRIXBASE = "https://matrix.org/_matrix/client/api/v1/"
|
||||
MYUSERNAME = "@davetest:matrix.org"
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
#!/usr/bin/env python
|
||||
from __future__ import print_function
|
||||
from argparse import ArgumentParser
|
||||
|
||||
import json
|
||||
import requests
|
||||
import sys
|
||||
import urllib
|
||||
from argparse import ArgumentParser
|
||||
|
||||
import requests
|
||||
|
||||
try:
|
||||
raw_input
|
||||
|
||||
58
debian/changelog
vendored
58
debian/changelog
vendored
@@ -1,3 +1,61 @@
|
||||
matrix-synapse-py3 (1.19.0ubuntu1) UNRELEASED; urgency=medium
|
||||
|
||||
* Use Type=notify in systemd service
|
||||
|
||||
-- Dexter Chua <dec41@srcf.net> Wed, 26 Aug 2020 12:41:36 +0000
|
||||
|
||||
matrix-synapse-py3 (1.19.1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.19.1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Thu, 27 Aug 2020 10:50:19 +0100
|
||||
|
||||
matrix-synapse-py3 (1.19.0) stable; urgency=medium
|
||||
|
||||
[ Synapse Packaging team ]
|
||||
* New synapse release 1.19.0.
|
||||
|
||||
[ Aaron Raimist ]
|
||||
* Fix outdated documentation for SYNAPSE_CACHE_FACTOR
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Mon, 17 Aug 2020 14:06:42 +0100
|
||||
|
||||
matrix-synapse-py3 (1.18.0) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.18.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Thu, 30 Jul 2020 10:55:53 +0100
|
||||
|
||||
matrix-synapse-py3 (1.17.0) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.17.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Mon, 13 Jul 2020 10:20:31 +0100
|
||||
|
||||
matrix-synapse-py3 (1.16.1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.16.1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Fri, 10 Jul 2020 12:09:24 +0100
|
||||
|
||||
matrix-synapse-py3 (1.17.0rc1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.17.0rc1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Thu, 09 Jul 2020 16:53:12 +0100
|
||||
|
||||
matrix-synapse-py3 (1.16.0) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.16.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Wed, 08 Jul 2020 11:03:48 +0100
|
||||
|
||||
matrix-synapse-py3 (1.15.2) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.15.2.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Thu, 02 Jul 2020 10:34:00 -0400
|
||||
|
||||
matrix-synapse-py3 (1.15.1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.15.1.
|
||||
|
||||
2
debian/matrix-synapse.default
vendored
2
debian/matrix-synapse.default
vendored
@@ -1,2 +1,2 @@
|
||||
# Specify environment variables used when running Synapse
|
||||
# SYNAPSE_CACHE_FACTOR=1 (default)
|
||||
# SYNAPSE_CACHE_FACTOR=0.5 (default)
|
||||
|
||||
2
debian/matrix-synapse.service
vendored
2
debian/matrix-synapse.service
vendored
@@ -2,7 +2,7 @@
|
||||
Description=Synapse Matrix homeserver
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Type=notify
|
||||
User=matrix-synapse
|
||||
WorkingDirectory=/var/lib/matrix-synapse
|
||||
EnvironmentFile=/etc/default/matrix-synapse
|
||||
|
||||
27
debian/synctl.ronn
vendored
27
debian/synctl.ronn
vendored
@@ -46,19 +46,20 @@ Configuration file may be generated as follows:
|
||||
## ENVIRONMENT
|
||||
|
||||
* `SYNAPSE_CACHE_FACTOR`:
|
||||
Synapse's architecture is quite RAM hungry currently - a lot of
|
||||
recent room data and metadata is deliberately cached in RAM in
|
||||
order to speed up common requests. This will be improved in
|
||||
future, but for now the easiest way to either reduce the RAM usage
|
||||
(at the risk of slowing things down) is to set the
|
||||
SYNAPSE_CACHE_FACTOR environment variable. Roughly speaking, a
|
||||
SYNAPSE_CACHE_FACTOR of 1.0 will max out at around 3-4GB of
|
||||
resident memory - this is what we currently run the matrix.org
|
||||
on. The default setting is currently 0.1, which is probably around
|
||||
a ~700MB footprint. You can dial it down further to 0.02 if
|
||||
desired, which targets roughly ~512MB. Conversely you can dial it
|
||||
up if you need performance for lots of users and have a box with a
|
||||
lot of RAM.
|
||||
Synapse's architecture is quite RAM hungry currently - we deliberately
|
||||
cache a lot of recent room data and metadata in RAM in order to speed up
|
||||
common requests. We'll improve this in the future, but for now the easiest
|
||||
way to either reduce the RAM usage (at the risk of slowing things down)
|
||||
is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
|
||||
variable. The default is 0.5, which can be decreased to reduce RAM usage
|
||||
in memory constrained enviroments, or increased if performance starts to
|
||||
degrade.
|
||||
|
||||
However, degraded performance due to a low cache factor, common on
|
||||
machines with slow disks, often leads to explosions in memory use due
|
||||
backlogged requests. In this case, reducing the cache factor will make
|
||||
things worse. Instead, try increasing it drastically. 2.0 is a good
|
||||
starting value.
|
||||
|
||||
## COPYRIGHT
|
||||
|
||||
|
||||
@@ -16,34 +16,36 @@ ARG PYTHON_VERSION=3.7
|
||||
###
|
||||
### Stage 0: builder
|
||||
###
|
||||
FROM docker.io/python:${PYTHON_VERSION}-alpine3.11 as builder
|
||||
FROM docker.io/python:${PYTHON_VERSION}-slim as builder
|
||||
|
||||
# install the OS build deps
|
||||
RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
libffi-dev \
|
||||
libjpeg-dev \
|
||||
libpq-dev \
|
||||
libssl-dev \
|
||||
libwebp-dev \
|
||||
libxml++2.6-dev \
|
||||
libxslt1-dev \
|
||||
zlib1g-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN apk add \
|
||||
build-base \
|
||||
libffi-dev \
|
||||
libjpeg-turbo-dev \
|
||||
libressl-dev \
|
||||
libxslt-dev \
|
||||
linux-headers \
|
||||
postgresql-dev \
|
||||
zlib-dev
|
||||
|
||||
# build things which have slow build steps, before we copy synapse, so that
|
||||
# the layer can be cached.
|
||||
#
|
||||
# (we really just care about caching a wheel here, as the "pip install" below
|
||||
# will install them again.)
|
||||
|
||||
# Build dependencies that are not available as wheels, to speed up rebuilds
|
||||
RUN pip install --prefix="/install" --no-warn-script-location \
|
||||
cryptography \
|
||||
msgpack-python \
|
||||
pillow \
|
||||
pynacl
|
||||
frozendict \
|
||||
jaeger-client \
|
||||
opentracing \
|
||||
prometheus-client \
|
||||
psycopg2 \
|
||||
pycparser \
|
||||
pyrsistent \
|
||||
pyyaml \
|
||||
simplejson \
|
||||
threadloop \
|
||||
thrift
|
||||
|
||||
# now install synapse and all of the python deps to /install.
|
||||
|
||||
COPY synapse /synapse/synapse/
|
||||
COPY scripts /synapse/scripts/
|
||||
COPY MANIFEST.in README.rst setup.py synctl /synapse/
|
||||
@@ -55,19 +57,16 @@ RUN pip install --prefix="/install" --no-warn-script-location \
|
||||
### Stage 1: runtime
|
||||
###
|
||||
|
||||
FROM docker.io/python:${PYTHON_VERSION}-alpine3.11
|
||||
FROM docker.io/python:${PYTHON_VERSION}-slim
|
||||
|
||||
# xmlsec is required for saml support
|
||||
RUN apk add --no-cache --virtual .runtime_deps \
|
||||
libffi \
|
||||
libjpeg-turbo \
|
||||
libressl \
|
||||
libxslt \
|
||||
libpq \
|
||||
zlib \
|
||||
su-exec \
|
||||
tzdata \
|
||||
xmlsec
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
gosu \
|
||||
libjpeg62-turbo \
|
||||
libpq5 \
|
||||
libwebp6 \
|
||||
xmlsec1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY --from=builder /install /usr/local
|
||||
COPY ./docker/start.py /start.py
|
||||
@@ -78,3 +77,6 @@ VOLUME ["/data"]
|
||||
EXPOSE 8008/tcp 8009/tcp 8448/tcp
|
||||
|
||||
ENTRYPOINT ["/start.py"]
|
||||
|
||||
HEALTHCHECK --interval=1m --timeout=5s \
|
||||
CMD curl -fSs http://localhost:8008/health || exit 1
|
||||
|
||||
@@ -94,6 +94,21 @@ The following environment variables are supported in run mode:
|
||||
* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
|
||||
* `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`.
|
||||
|
||||
## Generating an (admin) user
|
||||
|
||||
After synapse is running, you may wish to create a user via `register_new_matrix_user`.
|
||||
|
||||
This requires a `registration_shared_secret` to be set in your config file. Synapse
|
||||
must be restarted to pick up this change.
|
||||
|
||||
You can then call the script:
|
||||
|
||||
```
|
||||
docker exec -it synapse register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml --help
|
||||
```
|
||||
|
||||
Remember to remove the `registration_shared_secret` and restart if you no-longer need it.
|
||||
|
||||
## TLS support
|
||||
|
||||
The default configuration exposes a single HTTP port: http://localhost:8008. It
|
||||
@@ -147,3 +162,32 @@ docker build -t matrixdotorg/synapse -f docker/Dockerfile .
|
||||
|
||||
You can choose to build a different docker image by changing the value of the `-f` flag to
|
||||
point to another Dockerfile.
|
||||
|
||||
## Disabling the healthcheck
|
||||
|
||||
If you are using a non-standard port or tls inside docker you can disable the healthcheck
|
||||
whilst running the above `docker run` commands.
|
||||
|
||||
```
|
||||
--no-healthcheck
|
||||
```
|
||||
## Setting custom healthcheck on docker run
|
||||
|
||||
If you wish to point the healthcheck at a different port with docker command, add the following
|
||||
|
||||
```
|
||||
--health-cmd 'curl -fSs http://localhost:1234/health'
|
||||
```
|
||||
|
||||
## Setting the healthcheck in docker-compose file
|
||||
|
||||
You can add the following to set a custom healthcheck in a docker compose file.
|
||||
You will need version >2.1 for this to work.
|
||||
|
||||
```
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fSs", "http://localhost:8008/health"]
|
||||
interval: 1m
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
@@ -4,16 +4,10 @@ formatters:
|
||||
precise:
|
||||
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
|
||||
|
||||
filters:
|
||||
context:
|
||||
(): synapse.logging.context.LoggingContextFilter
|
||||
request: ""
|
||||
|
||||
handlers:
|
||||
console:
|
||||
class: logging.StreamHandler
|
||||
formatter: precise
|
||||
filters: [context]
|
||||
|
||||
loggers:
|
||||
synapse.storage.SQL:
|
||||
|
||||
@@ -120,7 +120,7 @@ def generate_config_from_template(config_dir, config_path, environ, ownership):
|
||||
|
||||
if ownership is not None:
|
||||
subprocess.check_output(["chown", "-R", ownership, "/data"])
|
||||
args = ["su-exec", ownership] + args
|
||||
args = ["gosu", ownership] + args
|
||||
|
||||
subprocess.check_output(args)
|
||||
|
||||
@@ -172,8 +172,8 @@ def run_generate_config(environ, ownership):
|
||||
# make sure that synapse has perms to write to the data dir.
|
||||
subprocess.check_output(["chown", ownership, data_dir])
|
||||
|
||||
args = ["su-exec", ownership] + args
|
||||
os.execv("/sbin/su-exec", args)
|
||||
args = ["gosu", ownership] + args
|
||||
os.execv("/usr/sbin/gosu", args)
|
||||
else:
|
||||
os.execv("/usr/local/bin/python", args)
|
||||
|
||||
@@ -189,7 +189,7 @@ def main(args, environ):
|
||||
ownership = "{}:{}".format(desired_uid, desired_gid)
|
||||
|
||||
if ownership is None:
|
||||
log("Will not perform chmod/su-exec as UserID already matches request")
|
||||
log("Will not perform chmod/gosu as UserID already matches request")
|
||||
|
||||
# In generate mode, generate a configuration and missing keys, then exit
|
||||
if mode == "generate":
|
||||
@@ -236,8 +236,8 @@ running with 'migrate_config'. See the README for more details.
|
||||
|
||||
args = ["python", "-m", synapse_worker, "--config-path", config_path]
|
||||
if ownership is not None:
|
||||
args = ["su-exec", ownership] + args
|
||||
os.execv("/sbin/su-exec", args)
|
||||
args = ["gosu", ownership] + args
|
||||
os.execv("/usr/sbin/gosu", args)
|
||||
else:
|
||||
os.execv("/usr/local/bin/python", args)
|
||||
|
||||
|
||||
@@ -10,5 +10,16 @@
|
||||
# homeserver.yaml. Instead, if you are starting from scratch, please generate
|
||||
# a fresh config using Synapse by following the instructions in INSTALL.md.
|
||||
|
||||
# Configuration options that take a time period can be set using a number
|
||||
# followed by a letter. Letters have the following meanings:
|
||||
# s = second
|
||||
# m = minute
|
||||
# h = hour
|
||||
# d = day
|
||||
# w = week
|
||||
# y = year
|
||||
# For example, setting redaction_retention_period: 5m would remove redacted
|
||||
# messages from the database after 5 minutes, rather than 5 months.
|
||||
|
||||
################################################################################
|
||||
|
||||
|
||||
@@ -12,13 +12,14 @@ introduced support for automatically provisioning certificates through
|
||||
In [March 2019](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430),
|
||||
Let's Encrypt announced that they were deprecating version 1 of the ACME
|
||||
protocol, with the plan to disable the use of it for new accounts in
|
||||
November 2019, and for existing accounts in June 2020.
|
||||
November 2019, for new domains in June 2020, and for existing accounts and
|
||||
domains in June 2021.
|
||||
|
||||
Synapse doesn't currently support version 2 of the ACME protocol, which
|
||||
means that:
|
||||
|
||||
* for existing installs, Synapse's built-in ACME support will continue
|
||||
to work until June 2020.
|
||||
to work until June 2021.
|
||||
* for new installs, this feature will not work at all.
|
||||
|
||||
Either way, it is recommended to move from Synapse's ACME support
|
||||
|
||||
@@ -5,6 +5,8 @@ This API will remove all trace of a room from your database.
|
||||
|
||||
All local users must have left the room before it can be removed.
|
||||
|
||||
See also: [Delete Room API](rooms.md#delete-room-api)
|
||||
|
||||
The API is:
|
||||
|
||||
```
|
||||
|
||||
@@ -318,3 +318,134 @@ Response:
|
||||
"state_events": 93534
|
||||
}
|
||||
```
|
||||
|
||||
# Room Members API
|
||||
|
||||
The Room Members admin API allows server admins to get a list of all members of a room.
|
||||
|
||||
The response includes the following fields:
|
||||
|
||||
* `members` - A list of all the members that are present in the room, represented by their ids.
|
||||
* `total` - Total number of members in the room.
|
||||
|
||||
## Usage
|
||||
|
||||
A standard request:
|
||||
|
||||
```
|
||||
GET /_synapse/admin/v1/rooms/<room_id>/members
|
||||
|
||||
{}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```
|
||||
{
|
||||
"members": [
|
||||
"@foo:matrix.org",
|
||||
"@bar:matrix.org",
|
||||
"@foobar:matrix.org
|
||||
],
|
||||
"total": 3
|
||||
}
|
||||
```
|
||||
|
||||
# Delete Room API
|
||||
|
||||
The Delete Room admin API allows server admins to remove rooms from server
|
||||
and block these rooms.
|
||||
It is a combination and improvement of "[Shutdown room](shutdown_room.md)"
|
||||
and "[Purge room](purge_room.md)" API.
|
||||
|
||||
Shuts down a room. Moves all local users and room aliases automatically to a
|
||||
new room if `new_room_user_id` is set. Otherwise local users only
|
||||
leave the room without any information.
|
||||
|
||||
The new room will be created with the user specified by the `new_room_user_id` parameter
|
||||
as room administrator and will contain a message explaining what happened. Users invited
|
||||
to the new room will have power level `-10` by default, and thus be unable to speak.
|
||||
|
||||
If `block` is `True` it prevents new joins to the old room.
|
||||
|
||||
This API will remove all trace of the old room from your database after removing
|
||||
all local users. If `purge` is `true` (the default), all traces of the old room will
|
||||
be removed from your database after removing all local users. If you do not want
|
||||
this to happen, set `purge` to `false`.
|
||||
Depending on the amount of history being purged a call to the API may take
|
||||
several minutes or longer.
|
||||
|
||||
The local server will only have the power to move local user and room aliases to
|
||||
the new room. Users on other servers will be unaffected.
|
||||
|
||||
The API is:
|
||||
|
||||
```json
|
||||
POST /_synapse/admin/v1/rooms/<room_id>/delete
|
||||
```
|
||||
|
||||
with a body of:
|
||||
```json
|
||||
{
|
||||
"new_room_user_id": "@someuser:example.com",
|
||||
"room_name": "Content Violation Notification",
|
||||
"message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
|
||||
"block": true,
|
||||
"purge": true
|
||||
}
|
||||
```
|
||||
|
||||
To use it, you will need to authenticate by providing an ``access_token`` for a
|
||||
server admin: see [README.rst](README.rst).
|
||||
|
||||
A response body like the following is returned:
|
||||
|
||||
```json
|
||||
{
|
||||
"kicked_users": [
|
||||
"@foobar:example.com"
|
||||
],
|
||||
"failed_to_kick_users": [],
|
||||
"local_aliases": [
|
||||
"#badroom:example.com",
|
||||
"#evilsaloon:example.com"
|
||||
],
|
||||
"new_room_id": "!newroomid:example.com"
|
||||
}
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
The following parameters should be set in the URL:
|
||||
|
||||
* `room_id` - The ID of the room.
|
||||
|
||||
The following JSON body parameters are available:
|
||||
|
||||
* `new_room_user_id` - Optional. If set, a new room will be created with this user ID
|
||||
as the creator and admin, and all users in the old room will be moved into that
|
||||
room. If not set, no new room will be created and the users will just be removed
|
||||
from the old room. The user ID must be on the local server, but does not necessarily
|
||||
have to belong to a registered user.
|
||||
* `room_name` - Optional. A string representing the name of the room that new users will be
|
||||
invited to. Defaults to `Content Violation Notification`
|
||||
* `message` - Optional. A string containing the first message that will be sent as
|
||||
`new_room_user_id` in the new room. Ideally this will clearly convey why the
|
||||
original room was shut down. Defaults to `Sharing illegal content on this server
|
||||
is not permitted and rooms in violation will be blocked.`
|
||||
* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
|
||||
future attempts to join the room. Defaults to `false`.
|
||||
* `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
|
||||
Defaults to `true`.
|
||||
|
||||
The JSON body must not be empty. The body must be at least `{}`.
|
||||
|
||||
## Response
|
||||
|
||||
The following fields are returned in the JSON response body:
|
||||
|
||||
* `kicked_users` - An array of users (`user_id`) that were kicked.
|
||||
* `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
|
||||
* `local_aliases` - An array of strings representing the local aliases that were migrated from
|
||||
the old room to the new.
|
||||
* `new_room_id` - A string representing the room ID of the new room.
|
||||
|
||||
@@ -10,6 +10,8 @@ disallow any further invites or joins.
|
||||
The local server will only have the power to move local user and room aliases to
|
||||
the new room. Users on other servers will be unaffected.
|
||||
|
||||
See also: [Delete Room API](rooms.md#delete-room-api)
|
||||
|
||||
## API
|
||||
|
||||
You will need to authenticate with an access token for an admin user.
|
||||
@@ -31,7 +33,7 @@ You will need to authenticate with an access token for an admin user.
|
||||
* `message` - Optional. A string containing the first message that will be sent as
|
||||
`new_room_user_id` in the new room. Ideally this will clearly convey why the
|
||||
original room was shut down.
|
||||
|
||||
|
||||
If not specified, the default value of `room_name` is "Content Violation
|
||||
Notification". The default value of `message` is "Sharing illegal content on
|
||||
othis server is not permitted and rooms in violation will be blocked."
|
||||
@@ -70,3 +72,30 @@ Response:
|
||||
"new_room_id": "!newroomid:example.com",
|
||||
},
|
||||
```
|
||||
|
||||
## Undoing room shutdowns
|
||||
|
||||
*Note*: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
|
||||
the structure can and does change without notice.
|
||||
|
||||
First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
|
||||
never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
|
||||
to recover at all:
|
||||
|
||||
* If the room was invite-only, your users will need to be re-invited.
|
||||
* If the room no longer has any members at all, it'll be impossible to rejoin.
|
||||
* The first user to rejoin will have to do so via an alias on a different server.
|
||||
|
||||
With all that being said, if you still want to try and recover the room:
|
||||
|
||||
1. For safety reasons, shut down Synapse.
|
||||
2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
|
||||
* For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
|
||||
* The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
|
||||
3. Restart Synapse.
|
||||
|
||||
You will have to manually handle, if you so choose, the following:
|
||||
|
||||
* Aliases that would have been redirected to the Content Violation room.
|
||||
* Users that would have been booted from the room (and will have been force-joined to the Content Violation room).
|
||||
* Removal of the Content Violation room if desired.
|
||||
|
||||
@@ -91,10 +91,14 @@ Body parameters:
|
||||
|
||||
- ``admin``, optional, defaults to ``false``.
|
||||
|
||||
- ``deactivated``, optional, defaults to ``false``.
|
||||
- ``deactivated``, optional. If unspecified, deactivation state will be left
|
||||
unchanged on existing accounts and set to ``false`` for new accounts.
|
||||
|
||||
If the user already exists then optional parameters default to the current value.
|
||||
|
||||
In order to re-activate an account ``deactivated`` must be set to ``false``. If
|
||||
users do not login via single-sign-on, a new ``password`` must be provided.
|
||||
|
||||
List Accounts
|
||||
=============
|
||||
|
||||
@@ -104,7 +108,7 @@ The api is::
|
||||
|
||||
GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
|
||||
|
||||
To use it, you will need to authenticate by providing an `access_token` for a
|
||||
To use it, you will need to authenticate by providing an ``access_token`` for a
|
||||
server admin: see `README.rst <README.rst>`_.
|
||||
|
||||
The parameter ``from`` is optional but used for pagination, denoting the
|
||||
@@ -115,8 +119,11 @@ from a previous call.
|
||||
The parameter ``limit`` is optional but is used for pagination, denoting the
|
||||
maximum number of items to return in this call. Defaults to ``100``.
|
||||
|
||||
The parameter ``user_id`` is optional and filters to only users with user IDs
|
||||
that contain this value.
|
||||
The parameter ``user_id`` is optional and filters to only return users with user IDs
|
||||
that contain this value. This parameter is ignored when using the ``name`` parameter.
|
||||
|
||||
The parameter ``name`` is optional and filters to only return users with user ID localparts
|
||||
**or** displaynames that contain this value.
|
||||
|
||||
The parameter ``guests`` is optional and if ``false`` will **exclude** guest users.
|
||||
Defaults to ``true`` to include guest users.
|
||||
@@ -207,9 +214,11 @@ Deactivate Account
|
||||
|
||||
This API deactivates an account. It removes active access tokens, resets the
|
||||
password, and deletes third-party IDs (to prevent the user requesting a
|
||||
password reset). It can also mark the user as GDPR-erased (stopping their data
|
||||
from distributed further, and deleting it entirely if there are no other
|
||||
references to it).
|
||||
password reset).
|
||||
|
||||
It can also mark the user as GDPR-erased. This means messages sent by the
|
||||
user will still be visible by anyone that was in the room when these messages
|
||||
were sent, but hidden from users joining the room afterwards.
|
||||
|
||||
The api is::
|
||||
|
||||
|
||||
@@ -47,6 +47,18 @@ you invite them to. This can be caused by an incorrectly-configured reverse
|
||||
proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions on how to correctly
|
||||
configure a reverse proxy.
|
||||
|
||||
### Known issues
|
||||
|
||||
**HTTP `308 Permanent Redirect` redirects are not followed**: Due to missing features
|
||||
in the HTTP library used by Synapse, 308 redirects are currently not followed by
|
||||
federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This
|
||||
may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`),
|
||||
and especially users of the Kubernetes *Nginx Ingress* module, which uses 308 redirect
|
||||
codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871)
|
||||
might be helpful. For other users, switching to a `301 Moved Permanently` code may be
|
||||
an option. 308 redirect codes will be supported properly in a future
|
||||
release of Synapse.
|
||||
|
||||
## Running a demo federation of Synapses
|
||||
|
||||
If you want to get up and running quickly with a trio of homeservers in a
|
||||
|
||||
97
docs/jwt.md
Normal file
97
docs/jwt.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# JWT Login Type
|
||||
|
||||
Synapse comes with a non-standard login type to support
|
||||
[JSON Web Tokens](https://en.wikipedia.org/wiki/JSON_Web_Token). In general the
|
||||
documentation for
|
||||
[the login endpoint](https://matrix.org/docs/spec/client_server/r0.6.1#login)
|
||||
is still valid (and the mechanism works similarly to the
|
||||
[token based login](https://matrix.org/docs/spec/client_server/r0.6.1#token-based)).
|
||||
|
||||
To log in using a JSON Web Token, clients should submit a `/login` request as
|
||||
follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "org.matrix.login.jwt",
|
||||
"token": "<jwt>"
|
||||
}
|
||||
```
|
||||
|
||||
Note that the login type of `m.login.jwt` is supported, but is deprecated. This
|
||||
will be removed in a future version of Synapse.
|
||||
|
||||
The `token` field should include the JSON web token with the following claims:
|
||||
|
||||
* The `sub` (subject) claim is required and should encode the local part of the
|
||||
user ID.
|
||||
* The expiration time (`exp`), not before time (`nbf`), and issued at (`iat`)
|
||||
claims are optional, but validated if present.
|
||||
* The issuer (`iss`) claim is optional, but required and validated if configured.
|
||||
* The audience (`aud`) claim is optional, but required and validated if configured.
|
||||
Providing the audience claim when not configured will cause validation to fail.
|
||||
|
||||
In the case that the token is not valid, the homeserver must respond with
|
||||
`403 Forbidden` and an error code of `M_FORBIDDEN`.
|
||||
|
||||
As with other login types, there are additional fields (e.g. `device_id` and
|
||||
`initial_device_display_name`) which can be included in the above request.
|
||||
|
||||
## Preparing Synapse
|
||||
|
||||
The JSON Web Token integration in Synapse uses the
|
||||
[`PyJWT`](https://pypi.org/project/pyjwt/) library, which must be installed
|
||||
as follows:
|
||||
|
||||
* The relevant libraries are included in the Docker images and Debian packages
|
||||
provided by `matrix.org` so no further action is needed.
|
||||
|
||||
* If you installed Synapse into a virtualenv, run `/path/to/env/bin/pip
|
||||
install synapse[pyjwt]` to install the necessary dependencies.
|
||||
|
||||
* For other installation mechanisms, see the documentation provided by the
|
||||
maintainer.
|
||||
|
||||
To enable the JSON web token integration, you should then add an `jwt_config` section
|
||||
to your configuration file (or uncomment the `enabled: true` line in the
|
||||
existing section). See [sample_config.yaml](./sample_config.yaml) for some
|
||||
sample settings.
|
||||
|
||||
## How to test JWT as a developer
|
||||
|
||||
Although JSON Web Tokens are typically generated from an external server, the
|
||||
examples below use [PyJWT](https://pyjwt.readthedocs.io/en/latest/) directly.
|
||||
|
||||
1. Configure Synapse with JWT logins, note that this example uses a pre-shared
|
||||
secret and an algorithm of HS256:
|
||||
|
||||
```yaml
|
||||
jwt_config:
|
||||
enabled: true
|
||||
secret: "my-secret-token"
|
||||
algorithm: "HS256"
|
||||
```
|
||||
2. Generate a JSON web token:
|
||||
|
||||
```bash
|
||||
$ pyjwt --key=my-secret-token --alg=HS256 encode sub=test-user
|
||||
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc
|
||||
```
|
||||
3. Query for the login types and ensure `org.matrix.login.jwt` is there:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/_matrix/client/r0/login
|
||||
```
|
||||
4. Login used the generated JSON web token from above:
|
||||
|
||||
```bash
|
||||
$ curl http://localhost:8082/_matrix/client/r0/login -X POST \
|
||||
--data '{"type":"org.matrix.login.jwt","token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc"}'
|
||||
{
|
||||
"access_token": "<access token>",
|
||||
"device_id": "ACBDEFGHI",
|
||||
"home_server": "localhost:8080",
|
||||
"user_id": "@test-user:localhost:8480"
|
||||
}
|
||||
```
|
||||
|
||||
You should now be able to use the returned access token to query the client API.
|
||||
@@ -27,7 +27,7 @@
|
||||
different thread to Synapse. This can make it more resilient to
|
||||
heavy load meaning metrics cannot be retrieved, and can be exposed
|
||||
to just internal networks easier. The served metrics are available
|
||||
over HTTP only, and will be available at `/`.
|
||||
over HTTP only, and will be available at `/_synapse/metrics`.
|
||||
|
||||
Add a new listener to homeserver.yaml:
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@ such as [Github][github-idp].
|
||||
[auth0]: https://auth0.com/
|
||||
[okta]: https://www.okta.com/
|
||||
[dex-idp]: https://github.com/dexidp/dex
|
||||
[keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols
|
||||
[hydra]: https://www.ory.sh/docs/hydra/
|
||||
[github-idp]: https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps
|
||||
|
||||
@@ -89,7 +90,50 @@ oidc_config:
|
||||
localpart_template: "{{ user.name }}"
|
||||
display_name_template: "{{ user.name|capitalize }}"
|
||||
```
|
||||
### [Keycloak][keycloak-idp]
|
||||
|
||||
[Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat.
|
||||
|
||||
Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to install Keycloak and set up a realm.
|
||||
|
||||
1. Click `Clients` in the sidebar and click `Create`
|
||||
|
||||
2. Fill in the fields as below:
|
||||
|
||||
| Field | Value |
|
||||
|-----------|-----------|
|
||||
| Client ID | `synapse` |
|
||||
| Client Protocol | `openid-connect` |
|
||||
|
||||
3. Click `Save`
|
||||
4. Fill in the fields as below:
|
||||
|
||||
| Field | Value |
|
||||
|-----------|-----------|
|
||||
| Client ID | `synapse` |
|
||||
| Enabled | `On` |
|
||||
| Client Protocol | `openid-connect` |
|
||||
| Access Type | `confidential` |
|
||||
| Valid Redirect URIs | `[synapse public baseurl]/_synapse/oidc/callback` |
|
||||
|
||||
5. Click `Save`
|
||||
6. On the Credentials tab, update the fields:
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Client Authenticator | `Client ID and Secret` |
|
||||
|
||||
7. Click `Regenerate Secret`
|
||||
8. Copy Secret
|
||||
|
||||
```yaml
|
||||
oidc_config:
|
||||
enabled: true
|
||||
issuer: "https://127.0.0.1:8443/auth/realms/{realm_name}"
|
||||
client_id: "synapse"
|
||||
client_secret: "copy secret generated from above"
|
||||
scopes: ["openid", "profile"]
|
||||
```
|
||||
### [Auth0][auth0]
|
||||
|
||||
1. Create a regular web application for Synapse
|
||||
|
||||
@@ -14,107 +14,109 @@ password auth provider module implementations:
|
||||
|
||||
* [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/)
|
||||
* [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth)
|
||||
* [matrix-synapse-rest-password-provider](https://github.com/ma1uta/matrix-synapse-rest-password-provider)
|
||||
|
||||
## Required methods
|
||||
|
||||
Password auth provider classes must provide the following methods:
|
||||
|
||||
*class* `SomeProvider.parse_config`(*config*)
|
||||
* `parse_config(config)`
|
||||
This method is passed the `config` object for this module from the
|
||||
homeserver configuration file.
|
||||
|
||||
> This method is passed the `config` object for this module from the
|
||||
> homeserver configuration file.
|
||||
>
|
||||
> It should perform any appropriate sanity checks on the provided
|
||||
> configuration, and return an object which is then passed into
|
||||
> `__init__`.
|
||||
It should perform any appropriate sanity checks on the provided
|
||||
configuration, and return an object which is then passed into
|
||||
|
||||
*class* `SomeProvider`(*config*, *account_handler*)
|
||||
This method should have the `@staticmethod` decoration.
|
||||
|
||||
> The constructor is passed the config object returned by
|
||||
> `parse_config`, and a `synapse.module_api.ModuleApi` object which
|
||||
> allows the password provider to check if accounts exist and/or create
|
||||
> new ones.
|
||||
* `__init__(self, config, account_handler)`
|
||||
|
||||
The constructor is passed the config object returned by
|
||||
`parse_config`, and a `synapse.module_api.ModuleApi` object which
|
||||
allows the password provider to check if accounts exist and/or create
|
||||
new ones.
|
||||
|
||||
## Optional methods
|
||||
|
||||
Password auth provider classes may optionally provide the following
|
||||
methods.
|
||||
Password auth provider classes may optionally provide the following methods:
|
||||
|
||||
*class* `SomeProvider.get_db_schema_files`()
|
||||
* `get_db_schema_files(self)`
|
||||
|
||||
> This method, if implemented, should return an Iterable of
|
||||
> `(name, stream)` pairs of database schema files. Each file is applied
|
||||
> in turn at initialisation, and a record is then made in the database
|
||||
> so that it is not re-applied on the next start.
|
||||
This method, if implemented, should return an Iterable of
|
||||
`(name, stream)` pairs of database schema files. Each file is applied
|
||||
in turn at initialisation, and a record is then made in the database
|
||||
so that it is not re-applied on the next start.
|
||||
|
||||
`someprovider.get_supported_login_types`()
|
||||
* `get_supported_login_types(self)`
|
||||
|
||||
> This method, if implemented, should return a `dict` mapping from a
|
||||
> login type identifier (such as `m.login.password`) to an iterable
|
||||
> giving the fields which must be provided by the user in the submission
|
||||
> to the `/login` api. These fields are passed in the `login_dict`
|
||||
> dictionary to `check_auth`.
|
||||
>
|
||||
> For example, if a password auth provider wants to implement a custom
|
||||
> login type of `com.example.custom_login`, where the client is expected
|
||||
> to pass the fields `secret1` and `secret2`, the provider should
|
||||
> implement this method and return the following dict:
|
||||
>
|
||||
> {"com.example.custom_login": ("secret1", "secret2")}
|
||||
This method, if implemented, should return a `dict` mapping from a
|
||||
login type identifier (such as `m.login.password`) to an iterable
|
||||
giving the fields which must be provided by the user in the submission
|
||||
to [the `/login` API](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login).
|
||||
These fields are passed in the `login_dict` dictionary to `check_auth`.
|
||||
|
||||
`someprovider.check_auth`(*username*, *login_type*, *login_dict*)
|
||||
For example, if a password auth provider wants to implement a custom
|
||||
login type of `com.example.custom_login`, where the client is expected
|
||||
to pass the fields `secret1` and `secret2`, the provider should
|
||||
implement this method and return the following dict:
|
||||
|
||||
> This method is the one that does the real work. If implemented, it
|
||||
> will be called for each login attempt where the login type matches one
|
||||
> of the keys returned by `get_supported_login_types`.
|
||||
>
|
||||
> It is passed the (possibly UNqualified) `user` provided by the client,
|
||||
> the login type, and a dictionary of login secrets passed by the
|
||||
> client.
|
||||
>
|
||||
> The method should return a Twisted `Deferred` object, which resolves
|
||||
> to the canonical `@localpart:domain` user id if authentication is
|
||||
> successful, and `None` if not.
|
||||
>
|
||||
> Alternatively, the `Deferred` can resolve to a `(str, func)` tuple, in
|
||||
> which case the second field is a callback which will be called with
|
||||
> the result from the `/login` call (including `access_token`,
|
||||
> `device_id`, etc.)
|
||||
```python
|
||||
{"com.example.custom_login": ("secret1", "secret2")}
|
||||
```
|
||||
|
||||
`someprovider.check_3pid_auth`(*medium*, *address*, *password*)
|
||||
* `check_auth(self, username, login_type, login_dict)`
|
||||
|
||||
> This method, if implemented, is called when a user attempts to
|
||||
> register or log in with a third party identifier, such as email. It is
|
||||
> passed the medium (ex. "email"), an address (ex.
|
||||
> "<jdoe@example.com>") and the user's password.
|
||||
>
|
||||
> The method should return a Twisted `Deferred` object, which resolves
|
||||
> to a `str` containing the user's (canonical) User ID if
|
||||
> authentication was successful, and `None` if not.
|
||||
>
|
||||
> As with `check_auth`, the `Deferred` may alternatively resolve to a
|
||||
> `(user_id, callback)` tuple.
|
||||
This method does the real work. If implemented, it
|
||||
will be called for each login attempt where the login type matches one
|
||||
of the keys returned by `get_supported_login_types`.
|
||||
|
||||
`someprovider.check_password`(*user_id*, *password*)
|
||||
It is passed the (possibly unqualified) `user` field provided by the client,
|
||||
the login type, and a dictionary of login secrets passed by the
|
||||
client.
|
||||
|
||||
> This method provides a simpler interface than
|
||||
> `get_supported_login_types` and `check_auth` for password auth
|
||||
> providers that just want to provide a mechanism for validating
|
||||
> `m.login.password` logins.
|
||||
>
|
||||
> Iif implemented, it will be called to check logins with an
|
||||
> `m.login.password` login type. It is passed a qualified
|
||||
> `@localpart:domain` user id, and the password provided by the user.
|
||||
>
|
||||
> The method should return a Twisted `Deferred` object, which resolves
|
||||
> to `True` if authentication is successful, and `False` if not.
|
||||
The method should return an `Awaitable` object, which resolves
|
||||
to the canonical `@localpart:domain` user ID if authentication is
|
||||
successful, and `None` if not.
|
||||
|
||||
`someprovider.on_logged_out`(*user_id*, *device_id*, *access_token*)
|
||||
Alternatively, the `Awaitable` can resolve to a `(str, func)` tuple, in
|
||||
which case the second field is a callback which will be called with
|
||||
the result from the `/login` call (including `access_token`,
|
||||
`device_id`, etc.)
|
||||
|
||||
> This method, if implemented, is called when a user logs out. It is
|
||||
> passed the qualified user ID, the ID of the deactivated device (if
|
||||
> any: access tokens are occasionally created without an associated
|
||||
> device ID), and the (now deactivated) access token.
|
||||
>
|
||||
> It may return a Twisted `Deferred` object; the logout request will
|
||||
> wait for the deferred to complete but the result is ignored.
|
||||
* `check_3pid_auth(self, medium, address, password)`
|
||||
|
||||
This method, if implemented, is called when a user attempts to
|
||||
register or log in with a third party identifier, such as email. It is
|
||||
passed the medium (ex. "email"), an address (ex.
|
||||
"<jdoe@example.com>") and the user's password.
|
||||
|
||||
The method should return an `Awaitable` object, which resolves
|
||||
to a `str` containing the user's (canonical) User id if
|
||||
authentication was successful, and `None` if not.
|
||||
|
||||
As with `check_auth`, the `Awaitable` may alternatively resolve to a
|
||||
`(user_id, callback)` tuple.
|
||||
|
||||
* `check_password(self, user_id, password)`
|
||||
|
||||
This method provides a simpler interface than
|
||||
`get_supported_login_types` and `check_auth` for password auth
|
||||
providers that just want to provide a mechanism for validating
|
||||
`m.login.password` logins.
|
||||
|
||||
If implemented, it will be called to check logins with an
|
||||
`m.login.password` login type. It is passed a qualified
|
||||
`@localpart:domain` user id, and the password provided by the user.
|
||||
|
||||
The method should return an `Awaitable` object, which resolves
|
||||
to `True` if authentication is successful, and `False` if not.
|
||||
|
||||
* `on_logged_out(self, user_id, device_id, access_token)`
|
||||
|
||||
This method, if implemented, is called when a user logs out. It is
|
||||
passed the qualified user ID, the ID of the deactivated device (if
|
||||
any: access tokens are occasionally created without an associated
|
||||
device ID), and the (now deactivated) access token.
|
||||
|
||||
It may return an `Awaitable` object; the logout request will
|
||||
wait for the `Awaitable` to complete, but the result is ignored.
|
||||
|
||||
@@ -188,6 +188,9 @@ to do step 2.
|
||||
|
||||
It is safe to at any time kill the port script and restart it.
|
||||
|
||||
Note that the database may take up significantly more (25% - 100% more)
|
||||
space on disk after porting to Postgres.
|
||||
|
||||
### Using the port script
|
||||
|
||||
Firstly, shut down the currently running synapse server and copy its
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
It is recommended to put a reverse proxy such as
|
||||
[nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
|
||||
[Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html),
|
||||
[Caddy](https://caddyserver.com/docs/proxy) or
|
||||
[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy) or
|
||||
[HAProxy](https://www.haproxy.org/) in front of Synapse. One advantage
|
||||
of doing so is that it means that you can expose the default https port
|
||||
(443) to Matrix clients without needing to run Synapse with root
|
||||
@@ -38,6 +38,11 @@ the reverse proxy and the homeserver.
|
||||
server {
|
||||
listen 443 ssl;
|
||||
listen [::]:443 ssl;
|
||||
|
||||
# For the federation port
|
||||
listen 8448 ssl default_server;
|
||||
listen [::]:8448 ssl default_server;
|
||||
|
||||
server_name matrix.example.com;
|
||||
|
||||
location /_matrix {
|
||||
@@ -48,17 +53,6 @@ server {
|
||||
client_max_body_size 10M;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8448 ssl default_server;
|
||||
listen [::]:8448 ssl default_server;
|
||||
server_name example.com;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8008;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**NOTE**: Do not add a path after the port in `proxy_pass`, otherwise nginx will
|
||||
@@ -145,3 +139,10 @@ client IP addresses are recorded correctly.
|
||||
Having done so, you can then use `https://matrix.example.com` (instead
|
||||
of `https://matrix.example.com:8448`) as the "Custom server" when
|
||||
connecting to Synapse from a client.
|
||||
|
||||
|
||||
## Health check endpoint
|
||||
|
||||
Synapse exposes a health check endpoint for use by reverse proxies.
|
||||
Each configured HTTP listener has a `/health` endpoint which always returns
|
||||
200 OK (and doesn't get logged).
|
||||
|
||||
@@ -10,6 +10,17 @@
|
||||
# homeserver.yaml. Instead, if you are starting from scratch, please generate
|
||||
# a fresh config using Synapse by following the instructions in INSTALL.md.
|
||||
|
||||
# Configuration options that take a time period can be set using a number
|
||||
# followed by a letter. Letters have the following meanings:
|
||||
# s = second
|
||||
# m = minute
|
||||
# h = hour
|
||||
# d = day
|
||||
# w = week
|
||||
# y = year
|
||||
# For example, setting redaction_retention_period: 5m would remove redacted
|
||||
# messages from the database after 5 minutes, rather than 5 months.
|
||||
|
||||
################################################################################
|
||||
|
||||
# Configuration file for Synapse.
|
||||
@@ -102,7 +113,9 @@ pid_file: DATADIR/homeserver.pid
|
||||
#gc_thresholds: [700, 10, 10]
|
||||
|
||||
# Set the limit on the returned events in the timeline in the get
|
||||
# and sync operations. The default value is -1, means no upper limit.
|
||||
# and sync operations. The default value is 100. -1 means no upper limit.
|
||||
#
|
||||
# Uncomment the following to increase the limit to 5000.
|
||||
#
|
||||
#filter_timeline_limit: 5000
|
||||
|
||||
@@ -118,38 +131,6 @@ pid_file: DATADIR/homeserver.pid
|
||||
#
|
||||
#enable_search: false
|
||||
|
||||
# Restrict federation to the following whitelist of domains.
|
||||
# N.B. we recommend also firewalling your federation listener to limit
|
||||
# inbound federation traffic as early as possible, rather than relying
|
||||
# purely on this application-layer restriction. If not specified, the
|
||||
# default is to whitelist everything.
|
||||
#
|
||||
#federation_domain_whitelist:
|
||||
# - lon.example.com
|
||||
# - nyc.example.com
|
||||
# - syd.example.com
|
||||
|
||||
# Prevent federation requests from being sent to the following
|
||||
# blacklist IP address CIDR ranges. If this option is not specified, or
|
||||
# specified with an empty list, no ip range blacklist will be enforced.
|
||||
#
|
||||
# As of Synapse v1.4.0 this option also affects any outbound requests to identity
|
||||
# servers provided by user input.
|
||||
#
|
||||
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
|
||||
# listed here, since they correspond to unroutable addresses.)
|
||||
#
|
||||
federation_ip_range_blacklist:
|
||||
- '127.0.0.0/8'
|
||||
- '10.0.0.0/8'
|
||||
- '172.16.0.0/12'
|
||||
- '192.168.0.0/16'
|
||||
- '100.64.0.0/10'
|
||||
- '169.254.0.0/16'
|
||||
- '::1/128'
|
||||
- 'fe80::/64'
|
||||
- 'fc00::/7'
|
||||
|
||||
# List of ports that Synapse should listen on, their purpose and their
|
||||
# configuration.
|
||||
#
|
||||
@@ -178,7 +159,7 @@ federation_ip_range_blacklist:
|
||||
# names: a list of names of HTTP resources. See below for a list of
|
||||
# valid resource names.
|
||||
#
|
||||
# compress: set to true to enable HTTP comression for this resource.
|
||||
# compress: set to true to enable HTTP compression for this resource.
|
||||
#
|
||||
# additional_resources: Only valid for an 'http' listener. A map of
|
||||
# additional endpoints which should be loaded via dynamic modules.
|
||||
@@ -283,7 +264,7 @@ listeners:
|
||||
# number of monthly active users.
|
||||
#
|
||||
# 'limit_usage_by_mau' disables/enables monthly active user blocking. When
|
||||
# anabled and a limit is reached the server returns a 'ResourceLimitError'
|
||||
# enabled and a limit is reached the server returns a 'ResourceLimitError'
|
||||
# with error type Codes.RESOURCE_LIMIT_EXCEEDED
|
||||
#
|
||||
# 'max_mau_value' is the hard limit of monthly active users above which
|
||||
@@ -344,6 +325,10 @@ limit_remote_rooms:
|
||||
#
|
||||
#complexity_error: "This room is too complex."
|
||||
|
||||
# allow server admins to join complex rooms. Default is false.
|
||||
#
|
||||
#admins_can_join: true
|
||||
|
||||
# Whether to require a user to be in the room to add an alias to it.
|
||||
# Defaults to 'true'.
|
||||
#
|
||||
@@ -393,11 +378,10 @@ retention:
|
||||
# min_lifetime: 1d
|
||||
# max_lifetime: 1y
|
||||
|
||||
# Retention policy limits. If set, a user won't be able to send a
|
||||
# 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
|
||||
# that's not within this range. This is especially useful in closed federations,
|
||||
# in which server admins can make sure every federating server applies the same
|
||||
# rules.
|
||||
# Retention policy limits. If set, and the state of a room contains a
|
||||
# 'm.room.retention' event in its state which contains a 'min_lifetime' or a
|
||||
# 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
|
||||
# to these limits when running purge jobs.
|
||||
#
|
||||
#allowed_lifetime_min: 1d
|
||||
#allowed_lifetime_max: 1y
|
||||
@@ -423,12 +407,19 @@ retention:
|
||||
# (e.g. every 12h), but not want that purge to be performed by a job that's
|
||||
# iterating over every room it knows, which could be heavy on the server.
|
||||
#
|
||||
# If any purge job is configured, it is strongly recommended to have at least
|
||||
# a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
|
||||
# set, or one job without 'shortest_max_lifetime' and one job without
|
||||
# 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
|
||||
# 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
|
||||
# room's policy to these values is done after the policies are retrieved from
|
||||
# Synapse's database (which is done using the range specified in a purge job's
|
||||
# configuration).
|
||||
#
|
||||
#purge_jobs:
|
||||
# - shortest_max_lifetime: 1d
|
||||
# longest_max_lifetime: 3d
|
||||
# - longest_max_lifetime: 3d
|
||||
# interval: 12h
|
||||
# - shortest_max_lifetime: 3d
|
||||
# longest_max_lifetime: 1y
|
||||
# interval: 1d
|
||||
|
||||
# Inhibits the /requestToken endpoints from returning an error that might leak
|
||||
@@ -608,6 +599,39 @@ acme:
|
||||
|
||||
|
||||
|
||||
# Restrict federation to the following whitelist of domains.
|
||||
# N.B. we recommend also firewalling your federation listener to limit
|
||||
# inbound federation traffic as early as possible, rather than relying
|
||||
# purely on this application-layer restriction. If not specified, the
|
||||
# default is to whitelist everything.
|
||||
#
|
||||
#federation_domain_whitelist:
|
||||
# - lon.example.com
|
||||
# - nyc.example.com
|
||||
# - syd.example.com
|
||||
|
||||
# Prevent federation requests from being sent to the following
|
||||
# blacklist IP address CIDR ranges. If this option is not specified, or
|
||||
# specified with an empty list, no ip range blacklist will be enforced.
|
||||
#
|
||||
# As of Synapse v1.4.0 this option also affects any outbound requests to identity
|
||||
# servers provided by user input.
|
||||
#
|
||||
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
|
||||
# listed here, since they correspond to unroutable addresses.)
|
||||
#
|
||||
federation_ip_range_blacklist:
|
||||
- '127.0.0.0/8'
|
||||
- '10.0.0.0/8'
|
||||
- '172.16.0.0/12'
|
||||
- '192.168.0.0/16'
|
||||
- '100.64.0.0/10'
|
||||
- '169.254.0.0/16'
|
||||
- '::1/128'
|
||||
- 'fe80::/64'
|
||||
- 'fc00::/7'
|
||||
|
||||
|
||||
## Caching ##
|
||||
|
||||
# Caching can be configured through the following options.
|
||||
@@ -682,7 +706,7 @@ caches:
|
||||
#database:
|
||||
# name: psycopg2
|
||||
# args:
|
||||
# user: synapse
|
||||
# user: synapse_user
|
||||
# password: secretpassword
|
||||
# database: synapse
|
||||
# host: localhost
|
||||
@@ -728,6 +752,10 @@ log_config: "CONFDIR/SERVERNAME.log.config"
|
||||
# - one for ratelimiting redactions by room admins. If this is not explicitly
|
||||
# set then it uses the same ratelimiting as per rc_message. This is useful
|
||||
# to allow room admins to deal with abuse quickly.
|
||||
# - two for ratelimiting number of rooms a user can join, "local" for when
|
||||
# users are joining rooms the server is already in (this is cheap) vs
|
||||
# "remote" for when users are trying to join rooms not on the server (which
|
||||
# can be more expensive)
|
||||
#
|
||||
# The defaults are as shown below.
|
||||
#
|
||||
@@ -753,6 +781,14 @@ log_config: "CONFDIR/SERVERNAME.log.config"
|
||||
#rc_admin_redaction:
|
||||
# per_second: 1
|
||||
# burst_count: 50
|
||||
#
|
||||
#rc_joins:
|
||||
# local:
|
||||
# per_second: 0.1
|
||||
# burst_count: 3
|
||||
# remote:
|
||||
# per_second: 0.01
|
||||
# burst_count: 3
|
||||
|
||||
|
||||
# Ratelimiting settings for incoming federation
|
||||
@@ -1147,24 +1183,6 @@ account_validity:
|
||||
#
|
||||
#default_identity_server: https://matrix.org
|
||||
|
||||
# The list of identity servers trusted to verify third party
|
||||
# identifiers by this server.
|
||||
#
|
||||
# Also defines the ID server which will be called when an account is
|
||||
# deactivated (one will be picked arbitrarily).
|
||||
#
|
||||
# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
|
||||
# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
|
||||
# background migration script, informing itself that the identity server all of its
|
||||
# 3PIDs have been bound to is likely one of the below.
|
||||
#
|
||||
# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
|
||||
# it is now solely used for the purposes of the background migration script, and can be
|
||||
# removed once it has run.
|
||||
#trusted_third_party_id_servers:
|
||||
# - matrix.org
|
||||
# - vector.im
|
||||
|
||||
# Handle threepid (email/phone etc) registration and password resets through a set of
|
||||
# *trusted* identity servers. Note that this allows the configured identity server to
|
||||
# reset passwords for accounts!
|
||||
@@ -1215,7 +1233,11 @@ account_threepid_delegates:
|
||||
#enable_3pid_changes: false
|
||||
|
||||
# Users who register on this homeserver will automatically be joined
|
||||
# to these rooms
|
||||
# to these rooms.
|
||||
#
|
||||
# By default, any room aliases included in this list will be created
|
||||
# as a publicly joinable room when the first user registers for the
|
||||
# homeserver. This behaviour can be customised with the settings below.
|
||||
#
|
||||
#auto_join_rooms:
|
||||
# - "#example:example.com"
|
||||
@@ -1223,10 +1245,62 @@ account_threepid_delegates:
|
||||
# Where auto_join_rooms are specified, setting this flag ensures that the
|
||||
# the rooms exist by creating them when the first user on the
|
||||
# homeserver registers.
|
||||
#
|
||||
# By default the auto-created rooms are publicly joinable from any federated
|
||||
# server. Use the autocreate_auto_join_rooms_federated and
|
||||
# autocreate_auto_join_room_preset settings below to customise this behaviour.
|
||||
#
|
||||
# Setting to false means that if the rooms are not manually created,
|
||||
# users cannot be auto-joined since they do not exist.
|
||||
#
|
||||
#autocreate_auto_join_rooms: true
|
||||
# Defaults to true. Uncomment the following line to disable automatically
|
||||
# creating auto-join rooms.
|
||||
#
|
||||
#autocreate_auto_join_rooms: false
|
||||
|
||||
# Whether the auto_join_rooms that are auto-created are available via
|
||||
# federation. Only has an effect if autocreate_auto_join_rooms is true.
|
||||
#
|
||||
# Note that whether a room is federated cannot be modified after
|
||||
# creation.
|
||||
#
|
||||
# Defaults to true: the room will be joinable from other servers.
|
||||
# Uncomment the following to prevent users from other homeservers from
|
||||
# joining these rooms.
|
||||
#
|
||||
#autocreate_auto_join_rooms_federated: false
|
||||
|
||||
# The room preset to use when auto-creating one of auto_join_rooms. Only has an
|
||||
# effect if autocreate_auto_join_rooms is true.
|
||||
#
|
||||
# This can be one of "public_chat", "private_chat", or "trusted_private_chat".
|
||||
# If a value of "private_chat" or "trusted_private_chat" is used then
|
||||
# auto_join_mxid_localpart must also be configured.
|
||||
#
|
||||
# Defaults to "public_chat", meaning that the room is joinable by anyone, including
|
||||
# federated servers if autocreate_auto_join_rooms_federated is true (the default).
|
||||
# Uncomment the following to require an invitation to join these rooms.
|
||||
#
|
||||
#autocreate_auto_join_room_preset: private_chat
|
||||
|
||||
# The local part of the user id which is used to create auto_join_rooms if
|
||||
# autocreate_auto_join_rooms is true. If this is not provided then the
|
||||
# initial user account that registers will be used to create the rooms.
|
||||
#
|
||||
# The user id is also used to invite new users to any auto-join rooms which
|
||||
# are set to invite-only.
|
||||
#
|
||||
# It *must* be configured if autocreate_auto_join_room_preset is set to
|
||||
# "private_chat" or "trusted_private_chat".
|
||||
#
|
||||
# Note that this must be specified in order for new users to be correctly
|
||||
# invited to any auto-join rooms which have been set to invite-only (either
|
||||
# at the time of creation or subsequently).
|
||||
#
|
||||
# Note that, if the room already exists, this user must be joined and
|
||||
# have the appropriate permissions to invite new members.
|
||||
#
|
||||
#auto_join_mxid_localpart: system
|
||||
|
||||
# When auto_join_rooms is specified, setting this flag to false prevents
|
||||
# guest accounts from being automatically joined to the rooms.
|
||||
@@ -1459,7 +1533,7 @@ saml2_config:
|
||||
|
||||
# The lifetime of a SAML session. This defines how long a user has to
|
||||
# complete the authentication process, if allow_unsolicited is unset.
|
||||
# The default is 5 minutes.
|
||||
# The default is 15 minutes.
|
||||
#
|
||||
#saml_session_lifetime: 5m
|
||||
|
||||
@@ -1514,6 +1588,17 @@ saml2_config:
|
||||
#
|
||||
#grandfathered_mxid_source_attribute: upn
|
||||
|
||||
# It is possible to configure Synapse to only allow logins if SAML attributes
|
||||
# match particular values. The requirements can be listed under
|
||||
# `attribute_requirements` as shown below. All of the listed attributes must
|
||||
# match for the login to be permitted.
|
||||
#
|
||||
#attribute_requirements:
|
||||
# - attribute: userGroup
|
||||
# value: "staff"
|
||||
# - attribute: department
|
||||
# value: "sales"
|
||||
|
||||
# Directory in which Synapse will try to find the template files below.
|
||||
# If not set, default templates from within the Synapse package will be used.
|
||||
#
|
||||
@@ -1544,7 +1629,7 @@ saml2_config:
|
||||
# use an OpenID Connect Provider for authentication, instead of its internal
|
||||
# password database.
|
||||
#
|
||||
# See https://github.com/matrix-org/synapse/blob/master/openid.md.
|
||||
# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md.
|
||||
#
|
||||
oidc_config:
|
||||
# Uncomment the following to enable authorization against an OpenID Connect
|
||||
@@ -1753,12 +1838,60 @@ sso:
|
||||
#template_dir: "res/templates"
|
||||
|
||||
|
||||
# The JWT needs to contain a globally unique "sub" (subject) claim.
|
||||
# JSON web token integration. The following settings can be used to make
|
||||
# Synapse JSON web tokens for authentication, instead of its internal
|
||||
# password database.
|
||||
#
|
||||
# Each JSON Web Token needs to contain a "sub" (subject) claim, which is
|
||||
# used as the localpart of the mxid.
|
||||
#
|
||||
# Additionally, the expiration time ("exp"), not before time ("nbf"),
|
||||
# and issued at ("iat") claims are validated if present.
|
||||
#
|
||||
# Note that this is a non-standard login type and client support is
|
||||
# expected to be non-existant.
|
||||
#
|
||||
# See https://github.com/matrix-org/synapse/blob/master/docs/jwt.md.
|
||||
#
|
||||
#jwt_config:
|
||||
# enabled: true
|
||||
# secret: "a secret"
|
||||
# algorithm: "HS256"
|
||||
# Uncomment the following to enable authorization using JSON web
|
||||
# tokens. Defaults to false.
|
||||
#
|
||||
#enabled: true
|
||||
|
||||
# This is either the private shared secret or the public key used to
|
||||
# decode the contents of the JSON web token.
|
||||
#
|
||||
# Required if 'enabled' is true.
|
||||
#
|
||||
#secret: "provided-by-your-issuer"
|
||||
|
||||
# The algorithm used to sign the JSON web token.
|
||||
#
|
||||
# Supported algorithms are listed at
|
||||
# https://pyjwt.readthedocs.io/en/latest/algorithms.html
|
||||
#
|
||||
# Required if 'enabled' is true.
|
||||
#
|
||||
#algorithm: "provided-by-your-issuer"
|
||||
|
||||
# The issuer to validate the "iss" claim against.
|
||||
#
|
||||
# Optional, if provided the "iss" claim will be required and
|
||||
# validated for all JSON web tokens.
|
||||
#
|
||||
#issuer: "provided-by-your-issuer"
|
||||
|
||||
# A list of audiences to validate the "aud" claim against.
|
||||
#
|
||||
# Optional, if provided the "aud" claim will be required and
|
||||
# validated for all JSON web tokens.
|
||||
#
|
||||
# Note that if the "aud" claim is included in a JSON web token then
|
||||
# validation will fail without configuring audiences.
|
||||
#
|
||||
#audiences:
|
||||
# - "provided-by-your-issuer"
|
||||
|
||||
|
||||
password_config:
|
||||
@@ -1849,8 +1982,8 @@ email:
|
||||
#
|
||||
#notif_from: "Your Friendly %(app)s homeserver <noreply@example.com>"
|
||||
|
||||
# app_name defines the default value for '%(app)s' in notif_from. It
|
||||
# defaults to 'Matrix'.
|
||||
# app_name defines the default value for '%(app)s' in notif_from and email
|
||||
# subjects. It defaults to 'Matrix'.
|
||||
#
|
||||
#app_name: my_branded_matrix_server
|
||||
|
||||
@@ -1880,9 +2013,7 @@ email:
|
||||
# Directory in which Synapse will try to find the template files below.
|
||||
# If not set, default templates from within the Synapse package will be used.
|
||||
#
|
||||
# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
|
||||
# If you *do* uncomment it, you will need to make sure that all the templates
|
||||
# below are in the directory.
|
||||
# Do not uncomment this setting unless you want to customise the templates.
|
||||
#
|
||||
# Synapse will look for the following templates in this directory:
|
||||
#
|
||||
@@ -1919,6 +2050,73 @@ email:
|
||||
#
|
||||
#template_dir: "res/templates"
|
||||
|
||||
# Subjects to use when sending emails from Synapse.
|
||||
#
|
||||
# The placeholder '%(app)s' will be replaced with the value of the 'app_name'
|
||||
# setting above, or by a value dictated by the Matrix client application.
|
||||
#
|
||||
# If a subject isn't overridden in this configuration file, the value used as
|
||||
# its example will be used.
|
||||
#
|
||||
#subjects:
|
||||
|
||||
# Subjects for notification emails.
|
||||
#
|
||||
# On top of the '%(app)s' placeholder, these can use the following
|
||||
# placeholders:
|
||||
#
|
||||
# * '%(person)s', which will be replaced by the display name of the user(s)
|
||||
# that sent the message(s), e.g. "Alice and Bob".
|
||||
# * '%(room)s', which will be replaced by the name of the room the
|
||||
# message(s) have been sent to, e.g. "My super room".
|
||||
#
|
||||
# See the example provided for each setting to see which placeholder can be
|
||||
# used and how to use them.
|
||||
#
|
||||
# Subject to use to notify about one message from one or more user(s) in a
|
||||
# room which has a name.
|
||||
#message_from_person_in_room: "[%(app)s] You have a message on %(app)s from %(person)s in the %(room)s room..."
|
||||
#
|
||||
# Subject to use to notify about one message from one or more user(s) in a
|
||||
# room which doesn't have a name.
|
||||
#message_from_person: "[%(app)s] You have a message on %(app)s from %(person)s..."
|
||||
#
|
||||
# Subject to use to notify about multiple messages from one or more users in
|
||||
# a room which doesn't have a name.
|
||||
#messages_from_person: "[%(app)s] You have messages on %(app)s from %(person)s..."
|
||||
#
|
||||
# Subject to use to notify about multiple messages in a room which has a
|
||||
# name.
|
||||
#messages_in_room: "[%(app)s] You have messages on %(app)s in the %(room)s room..."
|
||||
#
|
||||
# Subject to use to notify about multiple messages in multiple rooms.
|
||||
#messages_in_room_and_others: "[%(app)s] You have messages on %(app)s in the %(room)s room and others..."
|
||||
#
|
||||
# Subject to use to notify about multiple messages from multiple persons in
|
||||
# multiple rooms. This is similar to the setting above except it's used when
|
||||
# the room in which the notification was triggered has no name.
|
||||
#messages_from_person_and_others: "[%(app)s] You have messages on %(app)s from %(person)s and others..."
|
||||
#
|
||||
# Subject to use to notify about an invite to a room which has a name.
|
||||
#invite_from_person_to_room: "[%(app)s] %(person)s has invited you to join the %(room)s room on %(app)s..."
|
||||
#
|
||||
# Subject to use to notify about an invite to a room which doesn't have a
|
||||
# name.
|
||||
#invite_from_person: "[%(app)s] %(person)s has invited you to chat on %(app)s..."
|
||||
|
||||
# Subject for emails related to account administration.
|
||||
#
|
||||
# On top of the '%(app)s' placeholder, these one can use the
|
||||
# '%(server_name)s' placeholder, which will be replaced by the value of the
|
||||
# 'server_name' setting in your Synapse configuration.
|
||||
#
|
||||
# Subject to use when sending a password reset email.
|
||||
#password_reset: "[%(server_name)s] Password reset"
|
||||
#
|
||||
# Subject to use when sending a verification email to assert an address's
|
||||
# ownership.
|
||||
#email_validation: "[%(server_name)s] Validate your email"
|
||||
|
||||
|
||||
# Password providers allow homeserver administrators to integrate
|
||||
# their Synapse installation with existing authentication methods
|
||||
@@ -1978,6 +2176,26 @@ spam_checker:
|
||||
# example_stop_events_from: ['@bad:example.com']
|
||||
|
||||
|
||||
## Rooms ##
|
||||
|
||||
# Controls whether locally-created rooms should be end-to-end encrypted by
|
||||
# default.
|
||||
#
|
||||
# Possible options are "all", "invite", and "off". They are defined as:
|
||||
#
|
||||
# * "all": any locally-created room
|
||||
# * "invite": any room created with the "private_chat" or "trusted_private_chat"
|
||||
# room creation presets
|
||||
# * "off": this option will take no effect
|
||||
#
|
||||
# The default value is "off".
|
||||
#
|
||||
# Note that this option will only affect rooms created after it is set. It
|
||||
# will also not affect rooms created by other servers.
|
||||
#
|
||||
#encryption_enabled_by_default_for_room_type: invite
|
||||
|
||||
|
||||
# Uncomment to allow non-server-admin users to create groups on this server
|
||||
#
|
||||
#enable_group_creation: true
|
||||
@@ -2209,3 +2427,57 @@ opentracing:
|
||||
#
|
||||
# logging:
|
||||
# false
|
||||
|
||||
|
||||
## Workers ##
|
||||
|
||||
# Disables sending of outbound federation transactions on the main process.
|
||||
# Uncomment if using a federation sender worker.
|
||||
#
|
||||
#send_federation: false
|
||||
|
||||
# It is possible to run multiple federation sender workers, in which case the
|
||||
# work is balanced across them.
|
||||
#
|
||||
# This configuration must be shared between all federation sender workers, and if
|
||||
# changed all federation sender workers must be stopped at the same time and then
|
||||
# started, to ensure that all instances are running with the same config (otherwise
|
||||
# events may be dropped).
|
||||
#
|
||||
#federation_sender_instances:
|
||||
# - federation_sender1
|
||||
|
||||
# When using workers this should be a map from `worker_name` to the
|
||||
# HTTP replication listener of the worker, if configured.
|
||||
#
|
||||
#instance_map:
|
||||
# worker1:
|
||||
# host: localhost
|
||||
# port: 8034
|
||||
|
||||
# Experimental: When using workers you can define which workers should
|
||||
# handle event persistence and typing notifications. Any worker
|
||||
# specified here must also be in the `instance_map`.
|
||||
#
|
||||
#stream_writers:
|
||||
# events: worker1
|
||||
# typing: worker1
|
||||
|
||||
|
||||
# Configuration for Redis when using workers. This *must* be enabled when
|
||||
# using workers (unless using old style direct TCP configuration).
|
||||
#
|
||||
redis:
|
||||
# Uncomment the below to enable Redis support.
|
||||
#
|
||||
#enabled: true
|
||||
|
||||
# Optional host and port to use to connect to redis. Defaults to
|
||||
# localhost and 6379
|
||||
#
|
||||
#host: localhost
|
||||
#port: 6379
|
||||
|
||||
# Optional password if configured on the Redis instance
|
||||
#
|
||||
#password: <secret_password>
|
||||
|
||||
@@ -11,24 +11,33 @@ formatters:
|
||||
precise:
|
||||
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
|
||||
|
||||
filters:
|
||||
context:
|
||||
(): synapse.logging.context.LoggingContextFilter
|
||||
request: ""
|
||||
|
||||
handlers:
|
||||
file:
|
||||
class: logging.handlers.RotatingFileHandler
|
||||
class: logging.handlers.TimedRotatingFileHandler
|
||||
formatter: precise
|
||||
filename: /var/log/matrix-synapse/homeserver.log
|
||||
maxBytes: 104857600
|
||||
backupCount: 10
|
||||
filters: [context]
|
||||
when: midnight
|
||||
backupCount: 3 # Does not include the current log file.
|
||||
encoding: utf8
|
||||
|
||||
# Default to buffering writes to log file for efficiency. This means that
|
||||
# will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
|
||||
# logs will still be flushed immediately.
|
||||
buffer:
|
||||
class: logging.handlers.MemoryHandler
|
||||
target: file
|
||||
# The capacity is the number of log lines that are buffered before
|
||||
# being written to disk. Increasing this will lead to better
|
||||
# performance, at the expensive of it taking longer for log lines to
|
||||
# be written to disk.
|
||||
capacity: 10
|
||||
flushLevel: 30 # Flush for WARNING logs as well
|
||||
|
||||
# A handler that writes logs to stderr. Unused by default, but can be used
|
||||
# instead of "buffer" and "file" in the logger handlers.
|
||||
console:
|
||||
class: logging.StreamHandler
|
||||
formatter: precise
|
||||
filters: [context]
|
||||
|
||||
loggers:
|
||||
synapse.storage.SQL:
|
||||
@@ -36,8 +45,23 @@ loggers:
|
||||
# information such as access tokens.
|
||||
level: INFO
|
||||
|
||||
twisted:
|
||||
# We send the twisted logging directly to the file handler,
|
||||
# to work around https://github.com/matrix-org/synapse/issues/3471
|
||||
# when using "buffer" logger. Use "console" to log to stderr instead.
|
||||
handlers: [file]
|
||||
propagate: false
|
||||
|
||||
root:
|
||||
level: INFO
|
||||
handlers: [file, console]
|
||||
|
||||
# Write logs to the `buffer` handler, which will buffer them together in memory,
|
||||
# then write them to a file.
|
||||
#
|
||||
# Replace "buffer" with "console" to log to stderr instead. (Note that you'll
|
||||
# also need to update the configuation for the `twisted` logger above, in
|
||||
# this case.)
|
||||
#
|
||||
handlers: [buffer]
|
||||
|
||||
disable_existing_loggers: false
|
||||
|
||||
32
docs/synctl_workers.md
Normal file
32
docs/synctl_workers.md
Normal file
@@ -0,0 +1,32 @@
|
||||
### Using synctl with workers
|
||||
|
||||
If you want to use `synctl` to manage your synapse processes, you will need to
|
||||
create an an additional configuration file for the main synapse process. That
|
||||
configuration should look like this:
|
||||
|
||||
```yaml
|
||||
worker_app: synapse.app.homeserver
|
||||
```
|
||||
|
||||
Additionally, each worker app must be configured with the name of a "pid file",
|
||||
to which it will write its process ID when it starts. For example, for a
|
||||
synchrotron, you might write:
|
||||
|
||||
```yaml
|
||||
worker_pid_file: /home/matrix/synapse/worker1.pid
|
||||
```
|
||||
|
||||
Finally, to actually run your worker-based synapse, you must pass synctl the `-a`
|
||||
commandline option to tell it to operate on all the worker configurations found
|
||||
in the given directory, e.g.:
|
||||
|
||||
synctl -a $CONFIG/workers start
|
||||
|
||||
Currently one should always restart all workers when restarting or upgrading
|
||||
synapse, unless you explicitly know it's safe not to. For instance, restarting
|
||||
synapse without restarting all the synchrotrons may result in broken typing
|
||||
notifications.
|
||||
|
||||
To manipulate a specific worker, you pass the -w option to synctl:
|
||||
|
||||
synctl -w $CONFIG/workers/worker1.yaml restart
|
||||
@@ -1,9 +1,14 @@
|
||||
[Unit]
|
||||
Description=Synapse %i
|
||||
AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
|
||||
|
||||
# This service should be restarted when the synapse target is restarted.
|
||||
PartOf=matrix-synapse.target
|
||||
|
||||
# if this is started at the same time as the main, let the main process start
|
||||
# first, to initialise the database schema.
|
||||
After=matrix-synapse.service
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
NotifyAccess=main
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
worker_app: synapse.app.federation_reader
|
||||
worker_name: federation_reader1
|
||||
|
||||
worker_replication_host: 127.0.0.1
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
|
||||
@@ -7,6 +7,6 @@ who are present in a publicly viewable room present on the server.
|
||||
|
||||
The directory info is stored in various tables, which can (typically after
|
||||
DB corruption) get stale or out of sync. If this happens, for now the
|
||||
solution to fix it is to execute the SQL [here](../synapse/storage/data_stores/main/schema/delta/53/user_dir_populate.sql)
|
||||
solution to fix it is to execute the SQL [here](../synapse/storage/databases/main/schema/delta/53/user_dir_populate.sql)
|
||||
and then restart synapse. This should then start a background task to
|
||||
flush the current tables and regenerate the directory.
|
||||
|
||||
513
docs/workers.md
513
docs/workers.md
@@ -1,10 +1,10 @@
|
||||
# Scaling synapse via workers
|
||||
|
||||
For small instances it recommended to run Synapse in monolith mode (the
|
||||
default). For larger instances where performance is a concern it can be helpful
|
||||
to split out functionality into multiple separate python processes. These
|
||||
processes are called 'workers', and are (eventually) intended to scale
|
||||
horizontally independently.
|
||||
For small instances it recommended to run Synapse in the default monolith mode.
|
||||
For larger instances where performance is a concern it can be helpful to split
|
||||
out functionality into multiple separate python processes. These processes are
|
||||
called 'workers', and are (eventually) intended to scale horizontally
|
||||
independently.
|
||||
|
||||
Synapse's worker support is under active development and subject to change as
|
||||
we attempt to rapidly scale ever larger Synapse instances. However we are
|
||||
@@ -16,69 +16,123 @@ workers only work with PostgreSQL-based Synapse deployments. SQLite should only
|
||||
be used for demo purposes and any admin considering workers should already be
|
||||
running PostgreSQL.
|
||||
|
||||
## Master/worker communication
|
||||
## Main process/worker communication
|
||||
|
||||
The workers communicate with the master process via a Synapse-specific protocol
|
||||
called 'replication' (analogous to MySQL- or Postgres-style database
|
||||
replication) which feeds a stream of relevant data from the master to the
|
||||
workers so they can be kept in sync with the master process and database state.
|
||||
The processes communicate with each other via a Synapse-specific protocol called
|
||||
'replication' (analogous to MySQL- or Postgres-style database replication) which
|
||||
feeds streams of newly written data between processes so they can be kept in
|
||||
sync with the database state.
|
||||
|
||||
Additionally, workers may make HTTP requests to the master, to send information
|
||||
in the other direction. Typically this is used for operations which need to
|
||||
wait for a reply - such as sending an event.
|
||||
When configured to do so, Synapse uses a
|
||||
[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication
|
||||
stream between all configured Synapse processes. Additionally, processes may
|
||||
make HTTP requests to each other, primarily for operations which need to wait
|
||||
for a reply ─ such as sending an event.
|
||||
|
||||
## Configuration
|
||||
Redis support was added in v1.13.0 with it becoming the recommended method in
|
||||
v1.18.0. It replaced the old direct TCP connections (which is deprecated as of
|
||||
v1.18.0) to the main process. With Redis, rather than all the workers connecting
|
||||
to the main process, all the workers and the main process connect to Redis,
|
||||
which relays replication commands between processes. This can give a significant
|
||||
cpu saving on the main process and will be a prerequisite for upcoming
|
||||
performance improvements.
|
||||
|
||||
See the [Architectural diagram](#architectural-diagram) section at the end for
|
||||
a visualisation of what this looks like.
|
||||
|
||||
|
||||
## Setting up workers
|
||||
|
||||
A Redis server is required to manage the communication between the processes.
|
||||
The Redis server should be installed following the normal procedure for your
|
||||
distribution (e.g. `apt install redis-server` on Debian). It is safe to use an
|
||||
existing Redis deployment if you have one.
|
||||
|
||||
Once installed, check that Redis is running and accessible from the host running
|
||||
Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing
|
||||
a response of `+PONG`.
|
||||
|
||||
The appropriate dependencies must also be installed for Synapse. If using a
|
||||
virtualenv, these can be installed with:
|
||||
|
||||
```sh
|
||||
pip install matrix-synapse[redis]
|
||||
```
|
||||
|
||||
Note that these dependencies are included when synapse is installed with `pip
|
||||
install matrix-synapse[all]`. They are also included in the debian packages from
|
||||
`matrix.org` and in the docker images at
|
||||
https://hub.docker.com/r/matrixdotorg/synapse/.
|
||||
|
||||
To make effective use of the workers, you will need to configure an HTTP
|
||||
reverse-proxy such as nginx or haproxy, which will direct incoming requests to
|
||||
the correct worker, or to the main synapse instance. Note that this includes
|
||||
requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md)
|
||||
for information on setting up a reverse proxy.
|
||||
the correct worker, or to the main synapse instance. See
|
||||
[reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse
|
||||
proxy.
|
||||
|
||||
When using workers, each worker process has its own configuration file which
|
||||
contains settings specific to that worker, such as the HTTP listener that it
|
||||
provides (if any), logging configuration, etc.
|
||||
|
||||
Normally, the worker processes are configured to read from a shared
|
||||
configuration file as well as the worker-specific configuration files. This
|
||||
makes it easier to keep common configuration settings synchronised across all
|
||||
the processes.
|
||||
|
||||
The main process is somewhat special in this respect: it does not normally
|
||||
need its own configuration file and can take all of its configuration from the
|
||||
shared configuration file.
|
||||
|
||||
|
||||
### Shared configuration
|
||||
|
||||
Normally, only a couple of changes are needed to make an existing configuration
|
||||
file suitable for use with workers. First, you need to enable an "HTTP replication
|
||||
listener" for the main process; and secondly, you need to enable redis-based
|
||||
replication. For example:
|
||||
|
||||
To enable workers, you need to add *two* replication listeners to the
|
||||
main Synapse configuration file (`homeserver.yaml`). For example:
|
||||
|
||||
```yaml
|
||||
# extend the existing `listeners` section. This defines the ports that the
|
||||
# main process will listen on.
|
||||
listeners:
|
||||
# The TCP replication port
|
||||
- port: 9092
|
||||
bind_address: '127.0.0.1'
|
||||
type: replication
|
||||
|
||||
# The HTTP replication port
|
||||
- port: 9093
|
||||
bind_address: '127.0.0.1'
|
||||
type: http
|
||||
resources:
|
||||
- names: [replication]
|
||||
|
||||
redis:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
Under **no circumstances** should these replication API listeners be exposed to
|
||||
the public internet; they have no authentication and are unencrypted.
|
||||
See the sample config for the full documentation of each option.
|
||||
|
||||
You should then create a set of configs for the various worker processes. Each
|
||||
worker configuration file inherits the configuration of the main homeserver
|
||||
configuration file. You can then override configuration specific to that
|
||||
worker, e.g. the HTTP listener that it provides (if any); logging
|
||||
configuration; etc. You should minimise the number of overrides though to
|
||||
maintain a usable config.
|
||||
Under **no circumstances** should the replication listener be exposed to the
|
||||
public internet; it has no authentication and is unencrypted.
|
||||
|
||||
|
||||
### Worker configuration
|
||||
|
||||
In the config file for each worker, you must specify the type of worker
|
||||
application (`worker_app`). The currently available worker applications are
|
||||
listed below. You must also specify the replication endpoints that it should
|
||||
talk to on the main synapse process. `worker_replication_host` should specify
|
||||
the host of the main synapse, `worker_replication_port` should point to the TCP
|
||||
replication listener port and `worker_replication_http_port` should point to
|
||||
the HTTP replication port.
|
||||
application (`worker_app`), and you should specify a unqiue name for the worker
|
||||
(`worker_name`). The currently available worker applications are listed below.
|
||||
You must also specify the HTTP replication endpoint that it should talk to on
|
||||
the main synapse process. `worker_replication_host` should specify the host of
|
||||
the main synapse and `worker_replication_http_port` should point to the HTTP
|
||||
replication port. If the worker will handle HTTP requests then the
|
||||
`worker_listeners` option should be set with a `http` listener, in the same way
|
||||
as the `listeners` option in the shared config.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
worker_app: synapse.app.synchrotron
|
||||
worker_app: synapse.app.generic_worker
|
||||
worker_name: worker1
|
||||
|
||||
# The replication listener on the synapse to talk to.
|
||||
# The replication listener on the main synapse process.
|
||||
worker_replication_host: 127.0.0.1
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
@@ -87,142 +141,43 @@ worker_listeners:
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
- federation
|
||||
|
||||
worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml
|
||||
worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml
|
||||
```
|
||||
|
||||
...is a full configuration for a synchrotron worker instance, which will expose a
|
||||
plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided
|
||||
by the main synapse.
|
||||
...is a full configuration for a generic worker instance, which will expose a
|
||||
plain HTTP endpoint on port 8083 separately serving various endpoints, e.g.
|
||||
`/sync`, which are listed below.
|
||||
|
||||
Obviously you should configure your reverse-proxy to route the relevant
|
||||
endpoints to the worker (`localhost:8083` in the above example).
|
||||
|
||||
|
||||
### Running Synapse with workers
|
||||
|
||||
Finally, you need to start your worker processes. This can be done with either
|
||||
`synctl` or your distribution's preferred service manager such as `systemd`. We
|
||||
recommend the use of `systemd` where available: for information on setting up
|
||||
`systemd` to start synapse workers, see
|
||||
[systemd-with-workers](systemd-with-workers). To use `synctl`, see below.
|
||||
[systemd-with-workers](systemd-with-workers). To use `synctl`, see
|
||||
[synctl_workers.md](synctl_workers.md).
|
||||
|
||||
### **Experimental** support for replication over redis
|
||||
|
||||
As of Synapse v1.13.0, it is possible to configure Synapse to send replication
|
||||
via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an
|
||||
alternative to direct TCP connections to the master: rather than all the
|
||||
workers connecting to the master, all the workers and the master connect to
|
||||
Redis, which relays replication commands between processes. This can give a
|
||||
significant cpu saving on the master and will be a prerequisite for upcoming
|
||||
performance improvements.
|
||||
|
||||
Note that this support is currently experimental; you may experience lost
|
||||
messages and similar problems! It is strongly recommended that admins setting
|
||||
up workers for the first time use direct TCP replication as above.
|
||||
|
||||
To configure Synapse to use Redis:
|
||||
|
||||
1. Install Redis following the normal procedure for your distribution - for
|
||||
example, on Debian, `apt install redis-server`. (It is safe to use an
|
||||
existing Redis deployment if you have one: we use a pub/sub stream named
|
||||
according to the `server_name` of your synapse server.)
|
||||
2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1
|
||||
localhost 6379` and get a response of `+PONG`.
|
||||
3. Install the python prerequisites. If you installed synapse into a
|
||||
virtualenv, this can be done with:
|
||||
```sh
|
||||
pip install matrix-synapse[redis]
|
||||
```
|
||||
The debian packages from matrix.org already include the required
|
||||
dependencies.
|
||||
4. Add config to the shared configuration (`homeserver.yaml`):
|
||||
```yaml
|
||||
redis:
|
||||
enabled: true
|
||||
```
|
||||
Optional parameters which can go alongside `enabled` are `host`, `port`,
|
||||
`password`. Normally none of these are required.
|
||||
5. Restart master and all workers.
|
||||
|
||||
Once redis replication is in use, `worker_replication_port` is redundant and
|
||||
can be removed from the worker configuration files. Similarly, the
|
||||
configuration for the `listener` for the TCP replication port can be removed
|
||||
from the main configuration file. Note that the HTTP replication port is
|
||||
still required.
|
||||
|
||||
### Using synctl
|
||||
|
||||
If you want to use `synctl` to manage your synapse processes, you will need to
|
||||
create an an additional configuration file for the master synapse process. That
|
||||
configuration should look like this:
|
||||
|
||||
```yaml
|
||||
worker_app: synapse.app.homeserver
|
||||
```
|
||||
|
||||
Additionally, each worker app must be configured with the name of a "pid file",
|
||||
to which it will write its process ID when it starts. For example, for a
|
||||
synchrotron, you might write:
|
||||
|
||||
```yaml
|
||||
worker_pid_file: /home/matrix/synapse/synchrotron.pid
|
||||
```
|
||||
|
||||
Finally, to actually run your worker-based synapse, you must pass synctl the `-a`
|
||||
commandline option to tell it to operate on all the worker configurations found
|
||||
in the given directory, e.g.:
|
||||
|
||||
synctl -a $CONFIG/workers start
|
||||
|
||||
Currently one should always restart all workers when restarting or upgrading
|
||||
synapse, unless you explicitly know it's safe not to. For instance, restarting
|
||||
synapse without restarting all the synchrotrons may result in broken typing
|
||||
notifications.
|
||||
|
||||
To manipulate a specific worker, you pass the -w option to synctl:
|
||||
|
||||
synctl -w $CONFIG/workers/synchrotron.yaml restart
|
||||
|
||||
## Available worker applications
|
||||
|
||||
### `synapse.app.pusher`
|
||||
### `synapse.app.generic_worker`
|
||||
|
||||
Handles sending push notifications to sygnal and email. Doesn't handle any
|
||||
REST endpoints itself, but you should set `start_pushers: False` in the
|
||||
shared configuration file to stop the main synapse sending these notifications.
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
|
||||
### `synapse.app.synchrotron`
|
||||
|
||||
The synchrotron handles `sync` requests from clients. In particular, it can
|
||||
handle REST endpoints matching the following regular expressions:
|
||||
This worker can handle API requests matching the following regular
|
||||
expressions:
|
||||
|
||||
# Sync requests
|
||||
^/_matrix/client/(v2_alpha|r0)/sync$
|
||||
^/_matrix/client/(api/v1|v2_alpha|r0)/events$
|
||||
^/_matrix/client/(api/v1|r0)/initialSync$
|
||||
^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$
|
||||
|
||||
The above endpoints should all be routed to the synchrotron worker by the
|
||||
reverse-proxy configuration.
|
||||
|
||||
It is possible to run multiple instances of the synchrotron to scale
|
||||
horizontally. In this case the reverse-proxy should be configured to
|
||||
load-balance across the instances, though it will be more efficient if all
|
||||
requests from a particular user are routed to a single instance. Extracting
|
||||
a userid from the access token is currently left as an exercise for the reader.
|
||||
|
||||
### `synapse.app.appservice`
|
||||
|
||||
Handles sending output traffic to Application Services. Doesn't handle any
|
||||
REST endpoints itself, but you should set `notify_appservices: False` in the
|
||||
shared configuration file to stop the main synapse sending these notifications.
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
|
||||
### `synapse.app.federation_reader`
|
||||
|
||||
Handles a subset of federation endpoints. In particular, it can handle REST
|
||||
endpoints matching the following regular expressions:
|
||||
|
||||
# Federation requests
|
||||
^/_matrix/federation/v1/event/
|
||||
^/_matrix/federation/v1/state/
|
||||
^/_matrix/federation/v1/state_ids/
|
||||
@@ -242,40 +197,145 @@ endpoints matching the following regular expressions:
|
||||
^/_matrix/federation/v1/event_auth/
|
||||
^/_matrix/federation/v1/exchange_third_party_invite/
|
||||
^/_matrix/federation/v1/user/devices/
|
||||
^/_matrix/federation/v1/send/
|
||||
^/_matrix/federation/v1/get_groups_publicised$
|
||||
^/_matrix/key/v2/query
|
||||
|
||||
# Inbound federation transaction request
|
||||
^/_matrix/federation/v1/send/
|
||||
|
||||
# Client API requests
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
|
||||
^/_matrix/client/versions$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/joined_groups$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/
|
||||
|
||||
# Registration/login requests
|
||||
^/_matrix/client/(api/v1|r0|unstable)/login$
|
||||
^/_matrix/client/(r0|unstable)/register$
|
||||
^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
|
||||
|
||||
# Event sending requests
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/join/
|
||||
^/_matrix/client/(api/v1|r0|unstable)/profile/
|
||||
|
||||
|
||||
Additionally, the following REST endpoints can be handled for GET requests:
|
||||
|
||||
^/_matrix/federation/v1/groups/
|
||||
|
||||
The above endpoints should all be routed to the federation_reader worker by the
|
||||
reverse-proxy configuration.
|
||||
Pagination requests can also be handled, but all requests for a given
|
||||
room must be routed to the same instance. Additionally, care must be taken to
|
||||
ensure that the purge history admin API is not used while pagination requests
|
||||
for the room are in flight:
|
||||
|
||||
The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single
|
||||
instance.
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
|
||||
|
||||
Note that `federation` must be added to the listener resources in the worker config:
|
||||
Note that a HTTP listener with `client` and `federation` resources must be
|
||||
configured in the `worker_listeners` option in the worker config.
|
||||
|
||||
|
||||
#### Load balancing
|
||||
|
||||
It is possible to run multiple instances of this worker app, with incoming requests
|
||||
being load-balanced between them by the reverse-proxy. However, different endpoints
|
||||
have different characteristics and so admins
|
||||
may wish to run multiple groups of workers handling different endpoints so that
|
||||
load balancing can be done in different ways.
|
||||
|
||||
For `/sync` and `/initialSync` requests it will be more efficient if all
|
||||
requests from a particular user are routed to a single instance. Extracting a
|
||||
user ID from the access token or `Authorization` header is currently left as an
|
||||
exercise for the reader. Admins may additionally wish to separate out `/sync`
|
||||
requests that have a `since` query parameter from those that don't (and
|
||||
`/initialSync`), as requests that don't are known as "initial sync" that happens
|
||||
when a user logs in on a new device and can be *very* resource intensive, so
|
||||
isolating these requests will stop them from interfering with other users ongoing
|
||||
syncs.
|
||||
|
||||
Federation and client requests can be balanced via simple round robin.
|
||||
|
||||
The inbound federation transaction request `^/_matrix/federation/v1/send/`
|
||||
should be balanced by source IP so that transactions from the same remote server
|
||||
go to the same process.
|
||||
|
||||
Registration/login requests can be handled separately purely to help ensure that
|
||||
unexpected load doesn't affect new logins and sign ups.
|
||||
|
||||
Finally, event sending requests can be balanced by the room ID in the URI (or
|
||||
the full URI, or even just round robin), the room ID is the path component after
|
||||
`/rooms/`. If there is a large bridge connected that is sending or may send lots
|
||||
of events, then a dedicated set of workers can be provisioned to limit the
|
||||
effects of bursts of events from that bridge on events sent by normal users.
|
||||
|
||||
#### Stream writers
|
||||
|
||||
Additionally, there is *experimental* support for moving writing of specific
|
||||
streams (such as events) off of the main process to a particular worker. (This
|
||||
is only supported with Redis-based replication.)
|
||||
|
||||
Currently support streams are `events` and `typing`.
|
||||
|
||||
To enable this, the worker must have a HTTP replication listener configured,
|
||||
have a `worker_name` and be listed in the `instance_map` config. For example to
|
||||
move event persistence off to a dedicated worker, the shared configuration would
|
||||
include:
|
||||
|
||||
```yaml
|
||||
worker_app: synapse.app.federation_reader
|
||||
...
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: <port>
|
||||
resources:
|
||||
- names:
|
||||
- federation
|
||||
instance_map:
|
||||
event_persister1:
|
||||
host: localhost
|
||||
port: 8034
|
||||
|
||||
stream_writers:
|
||||
events: event_persister1
|
||||
```
|
||||
|
||||
|
||||
### `synapse.app.pusher`
|
||||
|
||||
Handles sending push notifications to sygnal and email. Doesn't handle any
|
||||
REST endpoints itself, but you should set `start_pushers: False` in the
|
||||
shared configuration file to stop the main synapse sending push notifications.
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
|
||||
### `synapse.app.appservice`
|
||||
|
||||
Handles sending output traffic to Application Services. Doesn't handle any
|
||||
REST endpoints itself, but you should set `notify_appservices: False` in the
|
||||
shared configuration file to stop the main synapse sending appservice notifications.
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
|
||||
|
||||
### `synapse.app.federation_sender`
|
||||
|
||||
Handles sending federation traffic to other servers. Doesn't handle any
|
||||
REST endpoints itself, but you should set `send_federation: False` in the
|
||||
shared configuration file to stop the main synapse sending this traffic.
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
If running multiple federation senders then you must list each
|
||||
instance in the `federation_sender_instances` option by their `worker_name`.
|
||||
All instances must be stopped and started when adding or removing instances.
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
federation_sender_instances:
|
||||
- federation_sender1
|
||||
- federation_sender2
|
||||
```
|
||||
|
||||
### `synapse.app.media_repository`
|
||||
|
||||
@@ -307,47 +367,12 @@ expose the `media` resource. For example:
|
||||
- media
|
||||
```
|
||||
|
||||
Note this worker cannot be load-balanced: only one instance should be active.
|
||||
Note that if running multiple media repositories they must be on the same server
|
||||
and you must configure a single instance to run the background tasks, e.g.:
|
||||
|
||||
### `synapse.app.client_reader`
|
||||
|
||||
Handles client API endpoints. It can handle REST endpoints matching the
|
||||
following regular expressions:
|
||||
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/login$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
|
||||
^/_matrix/client/versions$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/joined_groups$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/
|
||||
|
||||
Additionally, the following REST endpoints can be handled for GET requests:
|
||||
|
||||
^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/groups/.*$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/account_data/
|
||||
^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/rooms/[^/]*/account_data/
|
||||
|
||||
Additionally, the following REST endpoints can be handled, but all requests must
|
||||
be routed to the same instance:
|
||||
|
||||
^/_matrix/client/(r0|unstable)/register$
|
||||
^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
|
||||
|
||||
Pagination requests can also be handled, but all requests with the same path
|
||||
room must be routed to the same instance. Additionally, care must be taken to
|
||||
ensure that the purge history admin API is not used while pagination requests
|
||||
for the room are in flight:
|
||||
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
|
||||
```yaml
|
||||
media_instance_running_background_jobs: "media-repository-1"
|
||||
```
|
||||
|
||||
### `synapse.app.user_dir`
|
||||
|
||||
@@ -383,15 +408,65 @@ file. For example:
|
||||
|
||||
worker_main_http_uri: http://127.0.0.1:8008
|
||||
|
||||
### `synapse.app.event_creator`
|
||||
### Historical apps
|
||||
|
||||
Handles some event creation. It can handle REST endpoints matching:
|
||||
*Note:* Historically there used to be more apps, however they have been
|
||||
amalgamated into a single `synapse.app.generic_worker` app. The remaining apps
|
||||
are ones that do specific processing unrelated to requests, e.g. the `pusher`
|
||||
that handles sending out push notifications for new events. The intention is for
|
||||
all these to be folded into the `generic_worker` app and to use config to define
|
||||
which processes handle the various proccessing such as push notifications.
|
||||
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/
|
||||
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
|
||||
^/_matrix/client/(api/v1|r0|unstable)/join/
|
||||
^/_matrix/client/(api/v1|r0|unstable)/profile/
|
||||
|
||||
It will create events locally and then send them on to the main synapse
|
||||
instance to be persisted and handled.
|
||||
## Migration from old config
|
||||
|
||||
There are two main independent changes that have been made: introducing Redis
|
||||
support and merging apps into `synapse.app.generic_worker`. Both these changes
|
||||
are backwards compatible and so no changes to the config are required, however
|
||||
server admins are encouraged to plan to migrate to Redis as the old style direct
|
||||
TCP replication config is deprecated.
|
||||
|
||||
To migrate to Redis add the `redis` config as above, and optionally remove the
|
||||
TCP `replication` listener from master and `worker_replication_port` from worker
|
||||
config.
|
||||
|
||||
To migrate apps to use `synapse.app.generic_worker` simply update the
|
||||
`worker_app` option in the worker configs, and where worker are started (e.g.
|
||||
in systemd service files, but not required for synctl).
|
||||
|
||||
|
||||
## Architectural diagram
|
||||
|
||||
The following shows an example setup using Redis and a reverse proxy:
|
||||
|
||||
```
|
||||
Clients & Federation
|
||||
|
|
||||
v
|
||||
+-----------+
|
||||
| |
|
||||
| Reverse |
|
||||
| Proxy |
|
||||
| |
|
||||
+-----------+
|
||||
| | |
|
||||
| | | HTTP requests
|
||||
+-------------------+ | +-----------+
|
||||
| +---+ |
|
||||
| | |
|
||||
v v v
|
||||
+--------------+ +--------------+ +--------------+ +--------------+
|
||||
| Main | | Generic | | Generic | | Event |
|
||||
| Process | | Worker 1 | | Worker 2 | | Persister |
|
||||
+--------------+ +--------------+ +--------------+ +--------------+
|
||||
^ ^ | ^ | | ^ | ^ ^
|
||||
| | | | | | | | | |
|
||||
| | | | | HTTP | | | | |
|
||||
| +----------+<--|---|---------+ | | | |
|
||||
| | +-------------|-->+----------+ |
|
||||
| | | |
|
||||
| | | |
|
||||
v v v v
|
||||
====================================================================
|
||||
Redis pub/sub channel
|
||||
```
|
||||
|
||||
63
mypy.ini
63
mypy.ini
@@ -1,11 +1,66 @@
|
||||
[mypy]
|
||||
namespace_packages = True
|
||||
plugins = mypy_zope:plugin
|
||||
plugins = mypy_zope:plugin, scripts-dev/mypy_synapse_plugin.py
|
||||
follow_imports = silent
|
||||
check_untyped_defs = True
|
||||
show_error_codes = True
|
||||
show_traceback = True
|
||||
mypy_path = stubs
|
||||
files =
|
||||
synapse/api,
|
||||
synapse/appservice,
|
||||
synapse/config,
|
||||
synapse/event_auth.py,
|
||||
synapse/events/builder.py,
|
||||
synapse/events/spamcheck.py,
|
||||
synapse/federation,
|
||||
synapse/handlers/auth.py,
|
||||
synapse/handlers/cas_handler.py,
|
||||
synapse/handlers/directory.py,
|
||||
synapse/handlers/events.py,
|
||||
synapse/handlers/federation.py,
|
||||
synapse/handlers/identity.py,
|
||||
synapse/handlers/initial_sync.py,
|
||||
synapse/handlers/message.py,
|
||||
synapse/handlers/oidc_handler.py,
|
||||
synapse/handlers/pagination.py,
|
||||
synapse/handlers/presence.py,
|
||||
synapse/handlers/room.py,
|
||||
synapse/handlers/room_member.py,
|
||||
synapse/handlers/room_member_worker.py,
|
||||
synapse/handlers/saml_handler.py,
|
||||
synapse/handlers/sync.py,
|
||||
synapse/handlers/ui_auth,
|
||||
synapse/http/federation/well_known_resolver.py,
|
||||
synapse/http/server.py,
|
||||
synapse/http/site.py,
|
||||
synapse/logging/,
|
||||
synapse/metrics,
|
||||
synapse/module_api,
|
||||
synapse/notifier.py,
|
||||
synapse/push/pusherpool.py,
|
||||
synapse/push/push_rule_evaluator.py,
|
||||
synapse/replication,
|
||||
synapse/rest,
|
||||
synapse/server.py,
|
||||
synapse/server_notices,
|
||||
synapse/spam_checker_api,
|
||||
synapse/state,
|
||||
synapse/storage/databases/main/stream.py,
|
||||
synapse/storage/databases/main/ui_auth.py,
|
||||
synapse/storage/database.py,
|
||||
synapse/storage/engines,
|
||||
synapse/storage/state.py,
|
||||
synapse/storage/util,
|
||||
synapse/streams,
|
||||
synapse/types.py,
|
||||
synapse/util/caches/descriptors.py,
|
||||
synapse/util/caches/stream_change_cache.py,
|
||||
synapse/util/metrics.py,
|
||||
tests/replication,
|
||||
tests/test_utils,
|
||||
tests/rest/client/v2_alpha/test_auth.py,
|
||||
tests/util/test_stream_change_cache.py
|
||||
|
||||
[mypy-pymacaroons.*]
|
||||
ignore_missing_imports = True
|
||||
@@ -78,3 +133,9 @@ ignore_missing_imports = True
|
||||
|
||||
[mypy-authlib.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-rust_python_jaeger_reporter.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-nacl.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
@@ -24,7 +24,6 @@ DISTS = (
|
||||
"debian:sid",
|
||||
"ubuntu:xenial",
|
||||
"ubuntu:bionic",
|
||||
"ubuntu:eoan",
|
||||
"ubuntu:focal",
|
||||
)
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
# A script which checks that an appropriate news file has been added on this
|
||||
# branch.
|
||||
|
||||
echo -e "+++ \033[32mChecking newsfragment\033[m"
|
||||
|
||||
set -e
|
||||
|
||||
# make sure that origin/develop is up to date
|
||||
@@ -16,6 +18,8 @@ pr="$BUILDKITE_PULL_REQUEST"
|
||||
if ! git diff --quiet FETCH_HEAD... -- debian; then
|
||||
if git diff --quiet FETCH_HEAD... -- debian/changelog; then
|
||||
echo "Updates to debian directory, but no update to the changelog." >&2
|
||||
echo "!! Please see the contributing guide for help writing your changelog entry:" >&2
|
||||
echo "https://github.com/matrix-org/synapse/blob/develop/CONTRIBUTING.md#debian-changelog" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
@@ -26,7 +30,12 @@ if ! git diff --name-only FETCH_HEAD... | grep -qv '^debian/'; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
tox -qe check-newsfragment
|
||||
# Print a link to the contributing guide if the user makes a mistake
|
||||
CONTRIBUTING_GUIDE_TEXT="!! Please see the contributing guide for help writing your changelog entry:
|
||||
https://github.com/matrix-org/synapse/blob/develop/CONTRIBUTING.md#changelog"
|
||||
|
||||
# If check-newsfragment returns a non-zero exit code, print the contributing guide and exit
|
||||
tox -qe check-newsfragment || (echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2 && exit 1)
|
||||
|
||||
echo
|
||||
echo "--------------------------"
|
||||
@@ -38,6 +47,7 @@ for f in `git diff --name-only FETCH_HEAD... -- changelog.d`; do
|
||||
lastchar=`tr -d '\n' < $f | tail -c 1`
|
||||
if [ $lastchar != '.' -a $lastchar != '!' ]; then
|
||||
echo -e "\e[31mERROR: newsfragment $f does not end with a '.' or '!'\e[39m" >&2
|
||||
echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -47,5 +57,6 @@ done
|
||||
|
||||
if [[ -n "$pr" && "$matched" -eq 0 ]]; then
|
||||
echo -e "\e[31mERROR: Did not find a news fragment with the right number: expected changelog.d/$pr.*.\e[39m" >&2
|
||||
echo -e "$CONTRIBUTING_GUIDE_TEXT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
34
scripts-dev/check_line_terminators.sh
Executable file
34
scripts-dev/check_line_terminators.sh
Executable file
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# This script checks that line terminators in all repository files (excluding
|
||||
# those in the .git directory) feature unix line terminators.
|
||||
#
|
||||
# Usage:
|
||||
#
|
||||
# ./check_line_terminators.sh
|
||||
#
|
||||
# The script will emit exit code 1 if any files that do not use unix line
|
||||
# terminators are found, 0 otherwise.
|
||||
|
||||
# cd to the root of the repository
|
||||
cd `dirname $0`/..
|
||||
|
||||
# Find and print files with non-unix line terminators
|
||||
if find . -path './.git/*' -prune -o -type f -print0 | xargs -0 grep -I -l $'\r$'; then
|
||||
echo -e '\e[31mERROR: found files with CRLF line endings. See above.\e[39m'
|
||||
exit 1
|
||||
fi
|
||||
@@ -2,9 +2,9 @@ import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import urllib2
|
||||
|
||||
import dns.resolver
|
||||
import urllib2
|
||||
from signedjson.key import decode_verify_key_bytes, write_signing_keys
|
||||
from signedjson.sign import verify_signed_json
|
||||
from unpaddedbase64 import decode_base64
|
||||
|
||||
@@ -21,11 +21,12 @@ import argparse
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
|
||||
from six.moves.urllib import parse as urlparse
|
||||
from typing import Any, Optional
|
||||
from urllib import parse as urlparse
|
||||
|
||||
import nacl.signing
|
||||
import requests
|
||||
import signedjson.types
|
||||
import srvlookup
|
||||
import yaml
|
||||
from requests.adapters import HTTPAdapter
|
||||
@@ -70,7 +71,9 @@ def encode_canonical_json(value):
|
||||
).encode("UTF-8")
|
||||
|
||||
|
||||
def sign_json(json_object, signing_key, signing_name):
|
||||
def sign_json(
|
||||
json_object: Any, signing_key: signedjson.types.SigningKey, signing_name: str
|
||||
) -> Any:
|
||||
signatures = json_object.pop("signatures", {})
|
||||
unsigned = json_object.pop("unsigned", None)
|
||||
|
||||
@@ -123,7 +126,14 @@ def read_signing_keys(stream):
|
||||
return keys
|
||||
|
||||
|
||||
def request_json(method, origin_name, origin_key, destination, path, content):
|
||||
def request(
|
||||
method: Optional[str],
|
||||
origin_name: str,
|
||||
origin_key: signedjson.types.SigningKey,
|
||||
destination: str,
|
||||
path: str,
|
||||
content: Optional[str],
|
||||
) -> requests.Response:
|
||||
if method is None:
|
||||
if content is None:
|
||||
method = "GET"
|
||||
@@ -160,11 +170,14 @@ def request_json(method, origin_name, origin_key, destination, path, content):
|
||||
if method == "POST":
|
||||
headers["Content-Type"] = "application/json"
|
||||
|
||||
result = s.request(
|
||||
method=method, url=dest, headers=headers, verify=False, data=content
|
||||
return s.request(
|
||||
method=method,
|
||||
url=dest,
|
||||
headers=headers,
|
||||
verify=False,
|
||||
data=content,
|
||||
stream=True,
|
||||
)
|
||||
sys.stderr.write("Status Code: %d\n" % (result.status_code,))
|
||||
return result.json()
|
||||
|
||||
|
||||
def main():
|
||||
@@ -223,7 +236,7 @@ def main():
|
||||
with open(args.signing_key_path) as f:
|
||||
key = read_signing_keys(f)[0]
|
||||
|
||||
result = request_json(
|
||||
result = request(
|
||||
args.method,
|
||||
args.server_name,
|
||||
key,
|
||||
@@ -232,7 +245,12 @@ def main():
|
||||
content=args.body,
|
||||
)
|
||||
|
||||
json.dump(result, sys.stdout)
|
||||
sys.stderr.write("Status Code: %d\n" % (result.status_code,))
|
||||
|
||||
for chunk in result.iter_content():
|
||||
# we write raw utf8 to stdout.
|
||||
sys.stdout.buffer.write(chunk)
|
||||
|
||||
print("")
|
||||
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ from synapse.storage.pdu import PduStore
|
||||
from synapse.storage.signatures import SignatureStore
|
||||
|
||||
|
||||
class Store(object):
|
||||
class Store:
|
||||
_get_pdu_tuples = PduStore.__dict__["_get_pdu_tuples"]
|
||||
_get_pdu_content_hashes_txn = SignatureStore.__dict__["_get_pdu_content_hashes_txn"]
|
||||
_get_prev_pdu_hashes_txn = SignatureStore.__dict__["_get_prev_pdu_hashes_txn"]
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
#
|
||||
# Runs linting scripts over the local Synapse checkout
|
||||
# isort - sorts import statements
|
||||
# flake8 - lints and finds mistakes
|
||||
# black - opinionated code formatter
|
||||
# flake8 - lints and finds mistakes
|
||||
|
||||
set -e
|
||||
|
||||
@@ -11,11 +11,11 @@ if [ $# -ge 1 ]
|
||||
then
|
||||
files=$*
|
||||
else
|
||||
files="synapse tests scripts-dev scripts"
|
||||
files="synapse tests scripts-dev scripts contrib synctl"
|
||||
fi
|
||||
|
||||
echo "Linting these locations: $files"
|
||||
isort -y -rc $files
|
||||
flake8 $files
|
||||
isort $files
|
||||
python3 -m black $files
|
||||
./scripts-dev/config-lint.sh
|
||||
flake8 $files
|
||||
|
||||
85
scripts-dev/mypy_synapse_plugin.py
Normal file
85
scripts-dev/mypy_synapse_plugin.py
Normal file
@@ -0,0 +1,85 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""This is a mypy plugin for Synpase to deal with some of the funky typing that
|
||||
can crop up, e.g the cache descriptors.
|
||||
"""
|
||||
|
||||
from typing import Callable, Optional
|
||||
|
||||
from mypy.plugin import MethodSigContext, Plugin
|
||||
from mypy.typeops import bind_self
|
||||
from mypy.types import CallableType
|
||||
|
||||
|
||||
class SynapsePlugin(Plugin):
|
||||
def get_method_signature_hook(
|
||||
self, fullname: str
|
||||
) -> Optional[Callable[[MethodSigContext], CallableType]]:
|
||||
if fullname.startswith(
|
||||
"synapse.util.caches.descriptors._CachedFunction.__call__"
|
||||
):
|
||||
return cached_function_method_signature
|
||||
return None
|
||||
|
||||
|
||||
def cached_function_method_signature(ctx: MethodSigContext) -> CallableType:
|
||||
"""Fixes the `_CachedFunction.__call__` signature to be correct.
|
||||
|
||||
It already has *almost* the correct signature, except:
|
||||
|
||||
1. the `self` argument needs to be marked as "bound"; and
|
||||
2. any `cache_context` argument should be removed.
|
||||
"""
|
||||
|
||||
# First we mark this as a bound function signature.
|
||||
signature = bind_self(ctx.default_signature)
|
||||
|
||||
# Secondly, we remove any "cache_context" args.
|
||||
#
|
||||
# Note: We should be only doing this if `cache_context=True` is set, but if
|
||||
# it isn't then the code will raise an exception when its called anyway, so
|
||||
# its not the end of the world.
|
||||
context_arg_index = None
|
||||
for idx, name in enumerate(signature.arg_names):
|
||||
if name == "cache_context":
|
||||
context_arg_index = idx
|
||||
break
|
||||
|
||||
if context_arg_index:
|
||||
arg_types = list(signature.arg_types)
|
||||
arg_types.pop(context_arg_index)
|
||||
|
||||
arg_names = list(signature.arg_names)
|
||||
arg_names.pop(context_arg_index)
|
||||
|
||||
arg_kinds = list(signature.arg_kinds)
|
||||
arg_kinds.pop(context_arg_index)
|
||||
|
||||
signature = signature.copy_modified(
|
||||
arg_types=arg_types, arg_names=arg_names, arg_kinds=arg_kinds,
|
||||
)
|
||||
|
||||
return signature
|
||||
|
||||
|
||||
def plugin(version: str):
|
||||
# This is the entry point of the plugin, and let's us deal with the fact
|
||||
# that the mypy plugin interface is *not* stable by looking at the version
|
||||
# string.
|
||||
#
|
||||
# However, since we pin the version of mypy Synapse uses in CI, we don't
|
||||
# really care.
|
||||
return SynapsePlugin
|
||||
@@ -40,7 +40,7 @@ class MockHomeserver(HomeServer):
|
||||
config.server_name, reactor=reactor, config=config, **kwargs
|
||||
)
|
||||
|
||||
self.version_string = "Synapse/"+get_version_string(synapse)
|
||||
self.version_string = "Synapse/" + get_version_string(synapse)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -86,7 +86,7 @@ if __name__ == "__main__":
|
||||
store = hs.get_datastore()
|
||||
|
||||
async def run_background_updates():
|
||||
await store.db.updates.run_background_updates(sleep=False)
|
||||
await store.db_pool.updates.run_background_updates(sleep=False)
|
||||
# Stop the reactor to exit the script once every background update is run.
|
||||
reactor.stop()
|
||||
|
||||
|
||||
@@ -23,8 +23,6 @@ import sys
|
||||
import time
|
||||
import traceback
|
||||
|
||||
from six import string_types
|
||||
|
||||
import yaml
|
||||
|
||||
from twisted.internet import defer, reactor
|
||||
@@ -37,30 +35,29 @@ from synapse.logging.context import (
|
||||
make_deferred_yieldable,
|
||||
run_in_background,
|
||||
)
|
||||
from synapse.storage.data_stores.main.client_ips import ClientIpBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.deviceinbox import (
|
||||
DeviceInboxBackgroundUpdateStore,
|
||||
)
|
||||
from synapse.storage.data_stores.main.devices import DeviceBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.events_bg_updates import (
|
||||
from synapse.storage.database import DatabasePool, make_conn
|
||||
from synapse.storage.databases.main.client_ips import ClientIpBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.deviceinbox import DeviceInboxBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.devices import DeviceBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.events_bg_updates import (
|
||||
EventsBackgroundUpdatesStore,
|
||||
)
|
||||
from synapse.storage.data_stores.main.media_repository import (
|
||||
from synapse.storage.databases.main.media_repository import (
|
||||
MediaRepositoryBackgroundUpdateStore,
|
||||
)
|
||||
from synapse.storage.data_stores.main.registration import (
|
||||
from synapse.storage.databases.main.registration import (
|
||||
RegistrationBackgroundUpdateStore,
|
||||
find_max_generated_user_id_localpart,
|
||||
)
|
||||
from synapse.storage.data_stores.main.room import RoomBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.roommember import RoomMemberBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.search import SearchBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.state import MainStateBackgroundUpdateStore
|
||||
from synapse.storage.data_stores.main.stats import StatsStore
|
||||
from synapse.storage.data_stores.main.user_directory import (
|
||||
from synapse.storage.databases.main.room import RoomBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.roommember import RoomMemberBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.search import SearchBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.state import MainStateBackgroundUpdateStore
|
||||
from synapse.storage.databases.main.stats import StatsStore
|
||||
from synapse.storage.databases.main.user_directory import (
|
||||
UserDirectoryBackgroundUpdateStore,
|
||||
)
|
||||
from synapse.storage.data_stores.state.bg_updates import StateBackgroundUpdateStore
|
||||
from synapse.storage.database import Database, make_conn
|
||||
from synapse.storage.databases.state.bg_updates import StateBackgroundUpdateStore
|
||||
from synapse.storage.engines import create_engine
|
||||
from synapse.storage.prepare_database import prepare_database
|
||||
from synapse.util import Clock
|
||||
@@ -91,6 +88,7 @@ BOOLEAN_COLUMNS = {
|
||||
"account_validity": ["email_sent"],
|
||||
"redactions": ["have_censored"],
|
||||
"room_stats_state": ["is_federatable"],
|
||||
"local_media_repository": ["safe_from_quarantine"],
|
||||
}
|
||||
|
||||
|
||||
@@ -129,6 +127,26 @@ APPEND_ONLY_TABLES = [
|
||||
]
|
||||
|
||||
|
||||
IGNORED_TABLES = {
|
||||
# We don't port these tables, as they're a faff and we can regenerate
|
||||
# them anyway.
|
||||
"user_directory",
|
||||
"user_directory_search",
|
||||
"user_directory_search_content",
|
||||
"user_directory_search_docsize",
|
||||
"user_directory_search_segdir",
|
||||
"user_directory_search_segments",
|
||||
"user_directory_search_stat",
|
||||
"user_directory_search_pos",
|
||||
"users_who_share_private_rooms",
|
||||
"users_in_public_room",
|
||||
# UI auth sessions have foreign keys so additional care needs to be taken,
|
||||
# the sessions are transient anyway, so ignore them.
|
||||
"ui_auth_sessions",
|
||||
"ui_auth_sessions_credentials",
|
||||
}
|
||||
|
||||
|
||||
# Error returned by the run function. Used at the top-level part of the script to
|
||||
# handle errors and return codes.
|
||||
end_error = None
|
||||
@@ -155,14 +173,14 @@ class Store(
|
||||
StatsStore,
|
||||
):
|
||||
def execute(self, f, *args, **kwargs):
|
||||
return self.db.runInteraction(f.__name__, f, *args, **kwargs)
|
||||
return self.db_pool.runInteraction(f.__name__, f, *args, **kwargs)
|
||||
|
||||
def execute_sql(self, sql, *args):
|
||||
def r(txn):
|
||||
txn.execute(sql, args)
|
||||
return txn.fetchall()
|
||||
|
||||
return self.db.runInteraction("execute_sql", r)
|
||||
return self.db_pool.runInteraction("execute_sql", r)
|
||||
|
||||
def insert_many_txn(self, txn, table, headers, rows):
|
||||
sql = "INSERT INTO %s (%s) VALUES (%s)" % (
|
||||
@@ -207,7 +225,7 @@ class Porter(object):
|
||||
async def setup_table(self, table):
|
||||
if table in APPEND_ONLY_TABLES:
|
||||
# It's safe to just carry on inserting.
|
||||
row = await self.postgres_store.db.simple_select_one(
|
||||
row = await self.postgres_store.db_pool.simple_select_one(
|
||||
table="port_from_sqlite3",
|
||||
keyvalues={"table_name": table},
|
||||
retcols=("forward_rowid", "backward_rowid"),
|
||||
@@ -224,7 +242,7 @@ class Porter(object):
|
||||
) = await self._setup_sent_transactions()
|
||||
backward_chunk = 0
|
||||
else:
|
||||
await self.postgres_store.db.simple_insert(
|
||||
await self.postgres_store.db_pool.simple_insert(
|
||||
table="port_from_sqlite3",
|
||||
values={
|
||||
"table_name": table,
|
||||
@@ -254,7 +272,7 @@ class Porter(object):
|
||||
|
||||
await self.postgres_store.execute(delete_all)
|
||||
|
||||
await self.postgres_store.db.simple_insert(
|
||||
await self.postgres_store.db_pool.simple_insert(
|
||||
table="port_from_sqlite3",
|
||||
values={"table_name": table, "forward_rowid": 1, "backward_rowid": 0},
|
||||
)
|
||||
@@ -291,21 +309,14 @@ class Porter(object):
|
||||
)
|
||||
return
|
||||
|
||||
if table in (
|
||||
"user_directory",
|
||||
"user_directory_search",
|
||||
"users_who_share_rooms",
|
||||
"users_in_pubic_room",
|
||||
):
|
||||
# We don't port these tables, as they're a faff and we can regenreate
|
||||
# them anyway.
|
||||
if table in IGNORED_TABLES:
|
||||
self.progress.update(table, table_size) # Mark table as done
|
||||
return
|
||||
|
||||
if table == "user_directory_stream_pos":
|
||||
# We need to make sure there is a single row, `(X, null), as that is
|
||||
# what synapse expects to be there.
|
||||
await self.postgres_store.db.simple_insert(
|
||||
await self.postgres_store.db_pool.simple_insert(
|
||||
table=table, values={"stream_id": None}
|
||||
)
|
||||
self.progress.update(table, table_size) # Mark table as done
|
||||
@@ -346,7 +357,7 @@ class Porter(object):
|
||||
|
||||
return headers, forward_rows, backward_rows
|
||||
|
||||
headers, frows, brows = await self.sqlite_store.db.runInteraction(
|
||||
headers, frows, brows = await self.sqlite_store.db_pool.runInteraction(
|
||||
"select", r
|
||||
)
|
||||
|
||||
@@ -362,7 +373,7 @@ class Porter(object):
|
||||
def insert(txn):
|
||||
self.postgres_store.insert_many_txn(txn, table, headers[1:], rows)
|
||||
|
||||
self.postgres_store.db.simple_update_one_txn(
|
||||
self.postgres_store.db_pool.simple_update_one_txn(
|
||||
txn,
|
||||
table="port_from_sqlite3",
|
||||
keyvalues={"table_name": table},
|
||||
@@ -400,7 +411,7 @@ class Porter(object):
|
||||
|
||||
return headers, rows
|
||||
|
||||
headers, rows = await self.sqlite_store.db.runInteraction("select", r)
|
||||
headers, rows = await self.sqlite_store.db_pool.runInteraction("select", r)
|
||||
|
||||
if rows:
|
||||
forward_chunk = rows[-1][0] + 1
|
||||
@@ -438,7 +449,7 @@ class Porter(object):
|
||||
],
|
||||
)
|
||||
|
||||
self.postgres_store.db.simple_update_one_txn(
|
||||
self.postgres_store.db_pool.simple_update_one_txn(
|
||||
txn,
|
||||
table="port_from_sqlite3",
|
||||
keyvalues={"table_name": "event_search"},
|
||||
@@ -481,7 +492,7 @@ class Porter(object):
|
||||
db_conn, allow_outdated_version=allow_outdated_version
|
||||
)
|
||||
prepare_database(db_conn, engine, config=self.hs_config)
|
||||
store = Store(Database(hs, db_config, engine), db_conn, hs)
|
||||
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)
|
||||
db_conn.commit()
|
||||
|
||||
return store
|
||||
@@ -489,7 +500,7 @@ class Porter(object):
|
||||
async def run_background_updates_on_postgres(self):
|
||||
# Manually apply all background updates on the PostgreSQL database.
|
||||
postgres_ready = (
|
||||
await self.postgres_store.db.updates.has_completed_background_updates()
|
||||
await self.postgres_store.db_pool.updates.has_completed_background_updates()
|
||||
)
|
||||
|
||||
if not postgres_ready:
|
||||
@@ -498,9 +509,9 @@ class Porter(object):
|
||||
self.progress.set_state("Running background updates on PostgreSQL")
|
||||
|
||||
while not postgres_ready:
|
||||
await self.postgres_store.db.updates.do_next_background_update(100)
|
||||
await self.postgres_store.db_pool.updates.do_next_background_update(100)
|
||||
postgres_ready = await (
|
||||
self.postgres_store.db.updates.has_completed_background_updates()
|
||||
self.postgres_store.db_pool.updates.has_completed_background_updates()
|
||||
)
|
||||
|
||||
async def run(self):
|
||||
@@ -521,7 +532,7 @@ class Porter(object):
|
||||
|
||||
# Check if all background updates are done, abort if not.
|
||||
updates_complete = (
|
||||
await self.sqlite_store.db.updates.has_completed_background_updates()
|
||||
await self.sqlite_store.db_pool.updates.has_completed_background_updates()
|
||||
)
|
||||
if not updates_complete:
|
||||
end_error = (
|
||||
@@ -563,22 +574,24 @@ class Porter(object):
|
||||
)
|
||||
|
||||
try:
|
||||
await self.postgres_store.db.runInteraction("alter_table", alter_table)
|
||||
await self.postgres_store.db_pool.runInteraction(
|
||||
"alter_table", alter_table
|
||||
)
|
||||
except Exception:
|
||||
# On Error Resume Next
|
||||
pass
|
||||
|
||||
await self.postgres_store.db.runInteraction(
|
||||
await self.postgres_store.db_pool.runInteraction(
|
||||
"create_port_table", create_port_table
|
||||
)
|
||||
|
||||
# Step 2. Get tables.
|
||||
self.progress.set_state("Fetching tables")
|
||||
sqlite_tables = await self.sqlite_store.db.simple_select_onecol(
|
||||
sqlite_tables = await self.sqlite_store.db_pool.simple_select_onecol(
|
||||
table="sqlite_master", keyvalues={"type": "table"}, retcol="name"
|
||||
)
|
||||
|
||||
postgres_tables = await self.postgres_store.db.simple_select_onecol(
|
||||
postgres_tables = await self.postgres_store.db_pool.simple_select_onecol(
|
||||
table="information_schema.tables",
|
||||
keyvalues={},
|
||||
retcol="distinct table_name",
|
||||
@@ -610,8 +623,10 @@ class Porter(object):
|
||||
)
|
||||
)
|
||||
|
||||
# Step 5. Do final post-processing
|
||||
# Step 5. Set up sequences
|
||||
self.progress.set_state("Setting up sequence generators")
|
||||
await self._setup_state_group_id_seq()
|
||||
await self._setup_user_id_seq()
|
||||
|
||||
self.progress.done()
|
||||
except Exception as e:
|
||||
@@ -635,7 +650,7 @@ class Porter(object):
|
||||
return bool(col)
|
||||
if isinstance(col, bytes):
|
||||
return bytearray(col)
|
||||
elif isinstance(col, string_types) and "\0" in col:
|
||||
elif isinstance(col, str) and "\0" in col:
|
||||
logger.warning(
|
||||
"DROPPING ROW: NUL value in table %s col %s: %r",
|
||||
table,
|
||||
@@ -677,7 +692,7 @@ class Porter(object):
|
||||
|
||||
return headers, [r for r in rows if r[ts_ind] < yesterday]
|
||||
|
||||
headers, rows = await self.sqlite_store.db.runInteraction("select", r)
|
||||
headers, rows = await self.sqlite_store.db_pool.runInteraction("select", r)
|
||||
|
||||
rows = self._convert_rows("sent_transactions", headers, rows)
|
||||
|
||||
@@ -710,7 +725,7 @@ class Porter(object):
|
||||
next_chunk = await self.sqlite_store.execute(get_start_id)
|
||||
next_chunk = max(max_inserted_rowid + 1, next_chunk)
|
||||
|
||||
await self.postgres_store.db.simple_insert(
|
||||
await self.postgres_store.db_pool.simple_insert(
|
||||
table="port_from_sqlite3",
|
||||
values={
|
||||
"table_name": "sent_transactions",
|
||||
@@ -779,7 +794,14 @@ class Porter(object):
|
||||
next_id = curr_id + 1
|
||||
txn.execute("ALTER SEQUENCE state_group_id_seq RESTART WITH %s", (next_id,))
|
||||
|
||||
return self.postgres_store.db.runInteraction("setup_state_group_id_seq", r)
|
||||
return self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
|
||||
|
||||
def _setup_user_id_seq(self):
|
||||
def r(txn):
|
||||
next_id = find_max_generated_user_id_localpart(txn) + 1
|
||||
txn.execute("ALTER SEQUENCE user_id_seq RESTART WITH %s", (next_id,))
|
||||
|
||||
return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
|
||||
|
||||
|
||||
##############################################
|
||||
|
||||
@@ -26,12 +26,11 @@ ignore=W503,W504,E203,E731,E501
|
||||
|
||||
[isort]
|
||||
line_length = 88
|
||||
not_skip = __init__.py
|
||||
sections=FUTURE,STDLIB,COMPAT,THIRDPARTY,TWISTED,FIRSTPARTY,TESTS,LOCALFOLDER
|
||||
default_section=THIRDPARTY
|
||||
known_first_party = synapse
|
||||
known_tests=tests
|
||||
known_compat = mock,six
|
||||
known_compat = mock
|
||||
known_twisted=twisted,OpenSSL
|
||||
multi_line_output=3
|
||||
include_trailing_comma=true
|
||||
|
||||
47
stubs/frozendict.pyi
Normal file
47
stubs/frozendict.pyi
Normal file
@@ -0,0 +1,47 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Stub for frozendict.
|
||||
|
||||
from typing import (
|
||||
Any,
|
||||
Hashable,
|
||||
Iterable,
|
||||
Iterator,
|
||||
Mapping,
|
||||
overload,
|
||||
Tuple,
|
||||
TypeVar,
|
||||
)
|
||||
|
||||
_KT = TypeVar("_KT", bound=Hashable) # Key type.
|
||||
_VT = TypeVar("_VT") # Value type.
|
||||
|
||||
class frozendict(Mapping[_KT, _VT]):
|
||||
@overload
|
||||
def __init__(self, **kwargs: _VT) -> None: ...
|
||||
@overload
|
||||
def __init__(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
|
||||
@overload
|
||||
def __init__(
|
||||
self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT
|
||||
) -> None: ...
|
||||
def __getitem__(self, key: _KT) -> _VT: ...
|
||||
def __contains__(self, key: Any) -> bool: ...
|
||||
def copy(self, **add_or_replace: Any) -> frozendict: ...
|
||||
def __iter__(self) -> Iterator[_KT]: ...
|
||||
def __len__(self) -> int: ...
|
||||
def __repr__(self) -> str: ...
|
||||
def __hash__(self) -> int: ...
|
||||
@@ -22,6 +22,7 @@ class RedisProtocol:
|
||||
def publish(self, channel: str, message: bytes): ...
|
||||
|
||||
class SubscriberProtocol:
|
||||
def __init__(self, *args, **kwargs): ...
|
||||
password: Optional[str]
|
||||
def subscribe(self, channels: Union[str, List[str]]): ...
|
||||
def connectionMade(self): ...
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
""" This is a reference implementation of a Matrix homeserver.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
@@ -25,6 +26,9 @@ if sys.version_info < (3, 5):
|
||||
print("Synapse requires Python 3.5 or above.")
|
||||
sys.exit(1)
|
||||
|
||||
# Twisted and canonicaljson will fail to import when this file is executed to
|
||||
# get the __version__ during a fresh install. That's OK and subsequent calls to
|
||||
# actually start Synapse will import these libraries fine.
|
||||
try:
|
||||
from twisted.internet import protocol
|
||||
from twisted.internet.protocol import Factory
|
||||
@@ -36,7 +40,15 @@ try:
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
__version__ = "1.15.1"
|
||||
# Use the standard library json implementation instead of simplejson.
|
||||
try:
|
||||
from canonicaljson import set_json_library
|
||||
|
||||
set_json_library(json)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
__version__ = "1.20.0rc3"
|
||||
|
||||
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
|
||||
# We import here so that we don't have to install a bunch of deps when
|
||||
|
||||
@@ -23,8 +23,6 @@ import hmac
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from six.moves import input
|
||||
|
||||
import requests as _requests
|
||||
import yaml
|
||||
|
||||
|
||||
@@ -12,19 +12,14 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from six import itervalues
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
import pymacaroons
|
||||
from netaddr import IPAddress
|
||||
|
||||
from twisted.internet import defer
|
||||
from twisted.web.server import Request
|
||||
|
||||
import synapse.logging.opentracing as opentracing
|
||||
import synapse.types
|
||||
from synapse import event_auth
|
||||
from synapse.api.auth_blocking import AuthBlocking
|
||||
@@ -37,6 +32,7 @@ from synapse.api.errors import (
|
||||
)
|
||||
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
|
||||
from synapse.events import EventBase
|
||||
from synapse.logging import opentracing as opentracing
|
||||
from synapse.types import StateMap, UserID
|
||||
from synapse.util.caches import register_cache
|
||||
from synapse.util.caches.lrucache import LruCache
|
||||
@@ -62,7 +58,7 @@ class _InvalidMacaroonException(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Auth(object):
|
||||
class Auth:
|
||||
"""
|
||||
FIXME: This class contains a mix of functions for authenticating users
|
||||
of our client-server API and authenticating events added to room graphs.
|
||||
@@ -83,28 +79,28 @@ class Auth(object):
|
||||
self._track_appservice_user_ips = hs.config.track_appservice_user_ips
|
||||
self._macaroon_secret_key = hs.config.macaroon_secret_key
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def check_from_context(self, room_version: str, event, context, do_sig_check=True):
|
||||
prev_state_ids = yield context.get_prev_state_ids()
|
||||
auth_events_ids = yield self.compute_auth_events(
|
||||
async def check_from_context(
|
||||
self, room_version: str, event, context, do_sig_check=True
|
||||
):
|
||||
prev_state_ids = await context.get_prev_state_ids()
|
||||
auth_events_ids = self.compute_auth_events(
|
||||
event, prev_state_ids, for_verification=True
|
||||
)
|
||||
auth_events = yield self.store.get_events(auth_events_ids)
|
||||
auth_events = {(e.type, e.state_key): e for e in itervalues(auth_events)}
|
||||
auth_events = await self.store.get_events(auth_events_ids)
|
||||
auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
|
||||
|
||||
room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
|
||||
event_auth.check(
|
||||
room_version_obj, event, auth_events=auth_events, do_sig_check=do_sig_check
|
||||
)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def check_user_in_room(
|
||||
async def check_user_in_room(
|
||||
self,
|
||||
room_id: str,
|
||||
user_id: str,
|
||||
current_state: Optional[StateMap[EventBase]] = None,
|
||||
allow_departed_users: bool = False,
|
||||
):
|
||||
) -> EventBase:
|
||||
"""Check if the user is in the room, or was at some point.
|
||||
Args:
|
||||
room_id: The room to check.
|
||||
@@ -122,35 +118,35 @@ class Auth(object):
|
||||
Raises:
|
||||
AuthError if the user is/was not in the room.
|
||||
Returns:
|
||||
Deferred[Optional[EventBase]]:
|
||||
Membership event for the user if the user was in the
|
||||
room. This will be the join event if they are currently joined to
|
||||
the room. This will be the leave event if they have left the room.
|
||||
Membership event for the user if the user was in the
|
||||
room. This will be the join event if they are currently joined to
|
||||
the room. This will be the leave event if they have left the room.
|
||||
"""
|
||||
if current_state:
|
||||
member = current_state.get((EventTypes.Member, user_id), None)
|
||||
else:
|
||||
member = yield self.state.get_current_state(
|
||||
member = await self.state.get_current_state(
|
||||
room_id=room_id, event_type=EventTypes.Member, state_key=user_id
|
||||
)
|
||||
membership = member.membership if member else None
|
||||
|
||||
if membership == Membership.JOIN:
|
||||
return member
|
||||
if member:
|
||||
membership = member.membership
|
||||
|
||||
# XXX this looks totally bogus. Why do we not allow users who have been banned,
|
||||
# or those who were members previously and have been re-invited?
|
||||
if allow_departed_users and membership == Membership.LEAVE:
|
||||
forgot = yield self.store.did_forget(user_id, room_id)
|
||||
if not forgot:
|
||||
if membership == Membership.JOIN:
|
||||
return member
|
||||
|
||||
# XXX this looks totally bogus. Why do we not allow users who have been banned,
|
||||
# or those who were members previously and have been re-invited?
|
||||
if allow_departed_users and membership == Membership.LEAVE:
|
||||
forgot = await self.store.did_forget(user_id, room_id)
|
||||
if not forgot:
|
||||
return member
|
||||
|
||||
raise AuthError(403, "User %s not in room %s" % (user_id, room_id))
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def check_host_in_room(self, room_id, host):
|
||||
async def check_host_in_room(self, room_id, host):
|
||||
with Measure(self.clock, "check_host_in_room"):
|
||||
latest_event_ids = yield self.store.is_host_joined(room_id, host)
|
||||
latest_event_ids = await self.store.is_host_joined(room_id, host)
|
||||
return latest_event_ids
|
||||
|
||||
def can_federate(self, event, auth_events):
|
||||
@@ -161,14 +157,13 @@ class Auth(object):
|
||||
def get_public_keys(self, invite_event):
|
||||
return event_auth.get_public_keys(invite_event)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def get_user_by_req(
|
||||
async def get_user_by_req(
|
||||
self,
|
||||
request: Request,
|
||||
allow_guest: bool = False,
|
||||
rights: str = "access",
|
||||
allow_expired: bool = False,
|
||||
):
|
||||
) -> synapse.types.Requester:
|
||||
""" Get a registered user's ID.
|
||||
|
||||
Args:
|
||||
@@ -181,7 +176,7 @@ class Auth(object):
|
||||
/login will deliver access tokens regardless of expiration.
|
||||
|
||||
Returns:
|
||||
defer.Deferred: resolves to a `synapse.types.Requester` object
|
||||
Resolves to the requester
|
||||
Raises:
|
||||
InvalidClientCredentialsError if no user by that token exists or the token
|
||||
is invalid.
|
||||
@@ -195,14 +190,14 @@ class Auth(object):
|
||||
|
||||
access_token = self.get_access_token_from_request(request)
|
||||
|
||||
user_id, app_service = yield self._get_appservice_user_id(request)
|
||||
user_id, app_service = await self._get_appservice_user_id(request)
|
||||
if user_id:
|
||||
request.authenticated_entity = user_id
|
||||
opentracing.set_tag("authenticated_entity", user_id)
|
||||
opentracing.set_tag("appservice_id", app_service.id)
|
||||
|
||||
if ip_addr and self._track_appservice_user_ips:
|
||||
yield self.store.insert_client_ip(
|
||||
await self.store.insert_client_ip(
|
||||
user_id=user_id,
|
||||
access_token=access_token,
|
||||
ip=ip_addr,
|
||||
@@ -212,17 +207,18 @@ class Auth(object):
|
||||
|
||||
return synapse.types.create_requester(user_id, app_service=app_service)
|
||||
|
||||
user_info = yield self.get_user_by_access_token(
|
||||
user_info = await self.get_user_by_access_token(
|
||||
access_token, rights, allow_expired=allow_expired
|
||||
)
|
||||
user = user_info["user"]
|
||||
token_id = user_info["token_id"]
|
||||
is_guest = user_info["is_guest"]
|
||||
shadow_banned = user_info["shadow_banned"]
|
||||
|
||||
# Deny the request if the user account has expired.
|
||||
if self._account_validity.enabled and not allow_expired:
|
||||
user_id = user.to_string()
|
||||
expiration_ts = yield self.store.get_expiration_ts_for_user(user_id)
|
||||
expiration_ts = await self.store.get_expiration_ts_for_user(user_id)
|
||||
if (
|
||||
expiration_ts is not None
|
||||
and self.clock.time_msec() >= expiration_ts
|
||||
@@ -236,7 +232,7 @@ class Auth(object):
|
||||
device_id = user_info.get("device_id")
|
||||
|
||||
if user and access_token and ip_addr:
|
||||
yield self.store.insert_client_ip(
|
||||
await self.store.insert_client_ip(
|
||||
user_id=user.to_string(),
|
||||
access_token=access_token,
|
||||
ip=ip_addr,
|
||||
@@ -257,13 +253,17 @@ class Auth(object):
|
||||
opentracing.set_tag("device_id", device_id)
|
||||
|
||||
return synapse.types.create_requester(
|
||||
user, token_id, is_guest, device_id, app_service=app_service
|
||||
user,
|
||||
token_id,
|
||||
is_guest,
|
||||
shadow_banned,
|
||||
device_id,
|
||||
app_service=app_service,
|
||||
)
|
||||
except KeyError:
|
||||
raise MissingClientTokenError()
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _get_appservice_user_id(self, request):
|
||||
async def _get_appservice_user_id(self, request):
|
||||
app_service = self.store.get_app_service_by_token(
|
||||
self.get_access_token_from_request(request)
|
||||
)
|
||||
@@ -284,14 +284,13 @@ class Auth(object):
|
||||
|
||||
if not app_service.is_interested_in_user(user_id):
|
||||
raise AuthError(403, "Application service cannot masquerade as this user.")
|
||||
if not (yield self.store.get_user_by_id(user_id)):
|
||||
if not (await self.store.get_user_by_id(user_id)):
|
||||
raise AuthError(403, "Application service has not registered this user")
|
||||
return user_id, app_service
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def get_user_by_access_token(
|
||||
async def get_user_by_access_token(
|
||||
self, token: str, rights: str = "access", allow_expired: bool = False,
|
||||
):
|
||||
) -> dict:
|
||||
""" Validate access token and get user_id from it
|
||||
|
||||
Args:
|
||||
@@ -301,9 +300,10 @@ class Auth(object):
|
||||
allow_expired: If False, raises an InvalidClientTokenError
|
||||
if the token is expired
|
||||
Returns:
|
||||
Deferred[dict]: dict that includes:
|
||||
dict that includes:
|
||||
`user` (UserID)
|
||||
`is_guest` (bool)
|
||||
`shadow_banned` (bool)
|
||||
`token_id` (int|None): access token id. May be None if guest
|
||||
`device_id` (str|None): device corresponding to access token
|
||||
Raises:
|
||||
@@ -315,7 +315,7 @@ class Auth(object):
|
||||
|
||||
if rights == "access":
|
||||
# first look in the database
|
||||
r = yield self._look_up_user_by_access_token(token)
|
||||
r = await self._look_up_user_by_access_token(token)
|
||||
if r:
|
||||
valid_until_ms = r["valid_until_ms"]
|
||||
if (
|
||||
@@ -353,7 +353,7 @@ class Auth(object):
|
||||
# It would of course be much easier to store guest access
|
||||
# tokens in the database as well, but that would break existing
|
||||
# guest tokens.
|
||||
stored_user = yield self.store.get_user_by_id(user_id)
|
||||
stored_user = await self.store.get_user_by_id(user_id)
|
||||
if not stored_user:
|
||||
raise InvalidClientTokenError("Unknown user_id %s" % user_id)
|
||||
if not stored_user["is_guest"]:
|
||||
@@ -363,6 +363,7 @@ class Auth(object):
|
||||
ret = {
|
||||
"user": user,
|
||||
"is_guest": True,
|
||||
"shadow_banned": False,
|
||||
"token_id": None,
|
||||
# all guests get the same device id
|
||||
"device_id": GUEST_DEVICE_ID,
|
||||
@@ -372,6 +373,7 @@ class Auth(object):
|
||||
ret = {
|
||||
"user": user,
|
||||
"is_guest": False,
|
||||
"shadow_banned": False,
|
||||
"token_id": None,
|
||||
"device_id": None,
|
||||
}
|
||||
@@ -483,9 +485,8 @@ class Auth(object):
|
||||
now = self.hs.get_clock().time_msec()
|
||||
return now < expiry
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _look_up_user_by_access_token(self, token):
|
||||
ret = yield self.store.get_user_by_access_token(token)
|
||||
async def _look_up_user_by_access_token(self, token):
|
||||
ret = await self.store.get_user_by_access_token(token)
|
||||
if not ret:
|
||||
return None
|
||||
|
||||
@@ -496,6 +497,7 @@ class Auth(object):
|
||||
"user": UserID.from_string(ret.get("name")),
|
||||
"token_id": ret.get("token_id", None),
|
||||
"is_guest": False,
|
||||
"shadow_banned": ret.get("shadow_banned"),
|
||||
"device_id": ret.get("device_id"),
|
||||
"valid_until_ms": ret.get("valid_until_ms"),
|
||||
}
|
||||
@@ -508,7 +510,7 @@ class Auth(object):
|
||||
logger.warning("Unrecognised appservice access token.")
|
||||
raise InvalidClientTokenError()
|
||||
request.authenticated_entity = service.sender
|
||||
return defer.succeed(service)
|
||||
return service
|
||||
|
||||
async def is_server_admin(self, user: UserID) -> bool:
|
||||
""" Check if the given user is a local server admin.
|
||||
@@ -523,7 +525,7 @@ class Auth(object):
|
||||
|
||||
def compute_auth_events(
|
||||
self, event, current_state_ids: StateMap[str], for_verification: bool = False,
|
||||
):
|
||||
) -> List[str]:
|
||||
"""Given an event and current state return the list of event IDs used
|
||||
to auth an event.
|
||||
|
||||
@@ -531,16 +533,16 @@ class Auth(object):
|
||||
should be added to the event's `auth_events`.
|
||||
|
||||
Returns:
|
||||
defer.Deferred(list[str]): List of event IDs.
|
||||
List of event IDs.
|
||||
"""
|
||||
|
||||
if event.type == EventTypes.Create:
|
||||
return defer.succeed([])
|
||||
return []
|
||||
|
||||
# Currently we ignore the `for_verification` flag even though there are
|
||||
# some situations where we can drop particular auth events when adding
|
||||
# to the event's `auth_events` (e.g. joins pointing to previous joins
|
||||
# when room is publically joinable). Dropping event IDs has the
|
||||
# when room is publicly joinable). Dropping event IDs has the
|
||||
# advantage that the auth chain for the room grows slower, but we use
|
||||
# the auth chain in state resolution v2 to order events, which means
|
||||
# care must be taken if dropping events to ensure that it doesn't
|
||||
@@ -554,7 +556,7 @@ class Auth(object):
|
||||
if auth_ev_id:
|
||||
auth_ids.append(auth_ev_id)
|
||||
|
||||
return defer.succeed(auth_ids)
|
||||
return auth_ids
|
||||
|
||||
async def check_can_change_room_list(self, room_id: str, user: UserID):
|
||||
"""Determine whether the user is allowed to edit the room's entry in the
|
||||
@@ -637,10 +639,9 @@ class Auth(object):
|
||||
|
||||
return query_params[0].decode("ascii")
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def check_user_in_room_or_world_readable(
|
||||
async def check_user_in_room_or_world_readable(
|
||||
self, room_id: str, user_id: str, allow_departed_users: bool = False
|
||||
):
|
||||
) -> Tuple[str, Optional[str]]:
|
||||
"""Checks that the user is or was in the room or the room is world
|
||||
readable. If it isn't then an exception is raised.
|
||||
|
||||
@@ -651,10 +652,9 @@ class Auth(object):
|
||||
members but have now departed
|
||||
|
||||
Returns:
|
||||
Deferred[tuple[str, str|None]]: Resolves to the current membership of
|
||||
the user in the room and the membership event ID of the user. If
|
||||
the user is not in the room and never has been, then
|
||||
`(Membership.JOIN, None)` is returned.
|
||||
Resolves to the current membership of the user in the room and the
|
||||
membership event ID of the user. If the user is not in the room and
|
||||
never has been, then `(Membership.JOIN, None)` is returned.
|
||||
"""
|
||||
|
||||
try:
|
||||
@@ -663,12 +663,12 @@ class Auth(object):
|
||||
# * The user is a non-guest user, and was ever in the room
|
||||
# * The user is a guest user, and has joined the room
|
||||
# else it will throw.
|
||||
member_event = yield self.check_user_in_room(
|
||||
member_event = await self.check_user_in_room(
|
||||
room_id, user_id, allow_departed_users=allow_departed_users
|
||||
)
|
||||
return member_event.membership, member_event.event_id
|
||||
except AuthError:
|
||||
visibility = yield self.state.get_current_state(
|
||||
visibility = await self.state.get_current_state(
|
||||
room_id, EventTypes.RoomHistoryVisibility, ""
|
||||
)
|
||||
if (
|
||||
|
||||
@@ -15,8 +15,6 @@
|
||||
|
||||
import logging
|
||||
|
||||
from twisted.internet import defer
|
||||
|
||||
from synapse.api.constants import LimitBlockingTypes, UserTypes
|
||||
from synapse.api.errors import Codes, ResourceLimitError
|
||||
from synapse.config.server import is_threepid_reserved
|
||||
@@ -24,7 +22,7 @@ from synapse.config.server import is_threepid_reserved
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AuthBlocking(object):
|
||||
class AuthBlocking:
|
||||
def __init__(self, hs):
|
||||
self.store = hs.get_datastore()
|
||||
|
||||
@@ -36,8 +34,7 @@ class AuthBlocking(object):
|
||||
self._limit_usage_by_mau = hs.config.limit_usage_by_mau
|
||||
self._mau_limits_reserved_threepids = hs.config.mau_limits_reserved_threepids
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
|
||||
async def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
|
||||
"""Checks if the user should be rejected for some external reason,
|
||||
such as monthly active user limiting or global disable flag
|
||||
|
||||
@@ -60,7 +57,7 @@ class AuthBlocking(object):
|
||||
if user_id is not None:
|
||||
if user_id == self._server_notices_mxid:
|
||||
return
|
||||
if (yield self.store.is_support_user(user_id)):
|
||||
if await self.store.is_support_user(user_id):
|
||||
return
|
||||
|
||||
if self._hs_disabled:
|
||||
@@ -76,11 +73,11 @@ class AuthBlocking(object):
|
||||
|
||||
# If the user is already part of the MAU cohort or a trial user
|
||||
if user_id:
|
||||
timestamp = yield self.store.user_last_seen_monthly_active(user_id)
|
||||
timestamp = await self.store.user_last_seen_monthly_active(user_id)
|
||||
if timestamp:
|
||||
return
|
||||
|
||||
is_trial = yield self.store.is_trial_user(user_id)
|
||||
is_trial = await self.store.is_trial_user(user_id)
|
||||
if is_trial:
|
||||
return
|
||||
elif threepid:
|
||||
@@ -93,7 +90,7 @@ class AuthBlocking(object):
|
||||
# allow registration. Support users are excluded from MAU checks.
|
||||
return
|
||||
# Else if there is no room in the MAU bucket, bail
|
||||
current_mau = yield self.store.get_monthly_active_count()
|
||||
current_mau = await self.store.get_monthly_active_count()
|
||||
if current_mau >= self._max_mau_value:
|
||||
raise ResourceLimitError(
|
||||
403,
|
||||
|
||||
@@ -28,7 +28,7 @@ MAX_ALIAS_LENGTH = 255
|
||||
MAX_USERID_LENGTH = 255
|
||||
|
||||
|
||||
class Membership(object):
|
||||
class Membership:
|
||||
|
||||
"""Represents the membership states of a user in a room."""
|
||||
|
||||
@@ -40,7 +40,7 @@ class Membership(object):
|
||||
LIST = (INVITE, JOIN, KNOCK, LEAVE, BAN)
|
||||
|
||||
|
||||
class PresenceState(object):
|
||||
class PresenceState:
|
||||
"""Represents the presence state of a user."""
|
||||
|
||||
OFFLINE = "offline"
|
||||
@@ -48,14 +48,14 @@ class PresenceState(object):
|
||||
ONLINE = "online"
|
||||
|
||||
|
||||
class JoinRules(object):
|
||||
class JoinRules:
|
||||
PUBLIC = "public"
|
||||
KNOCK = "knock"
|
||||
INVITE = "invite"
|
||||
PRIVATE = "private"
|
||||
|
||||
|
||||
class LoginType(object):
|
||||
class LoginType:
|
||||
PASSWORD = "m.login.password"
|
||||
EMAIL_IDENTITY = "m.login.email.identity"
|
||||
MSISDN = "m.login.msisdn"
|
||||
@@ -65,7 +65,7 @@ class LoginType(object):
|
||||
DUMMY = "m.login.dummy"
|
||||
|
||||
|
||||
class EventTypes(object):
|
||||
class EventTypes:
|
||||
Member = "m.room.member"
|
||||
Create = "m.room.create"
|
||||
Tombstone = "m.room.tombstone"
|
||||
@@ -96,17 +96,17 @@ class EventTypes(object):
|
||||
Presence = "m.presence"
|
||||
|
||||
|
||||
class RejectedReason(object):
|
||||
class RejectedReason:
|
||||
AUTH_ERROR = "auth_error"
|
||||
|
||||
|
||||
class RoomCreationPreset(object):
|
||||
class RoomCreationPreset:
|
||||
PRIVATE_CHAT = "private_chat"
|
||||
PUBLIC_CHAT = "public_chat"
|
||||
TRUSTED_PRIVATE_CHAT = "trusted_private_chat"
|
||||
|
||||
|
||||
class ThirdPartyEntityKind(object):
|
||||
class ThirdPartyEntityKind:
|
||||
USER = "user"
|
||||
LOCATION = "location"
|
||||
|
||||
@@ -115,7 +115,7 @@ ServerNoticeMsgType = "m.server_notice"
|
||||
ServerNoticeLimitReached = "m.server_notice.usage_limit_reached"
|
||||
|
||||
|
||||
class UserTypes(object):
|
||||
class UserTypes:
|
||||
"""Allows for user type specific behaviour. With the benefit of hindsight
|
||||
'admin' and 'guest' users should also be UserTypes. Normal users are type None
|
||||
"""
|
||||
@@ -125,7 +125,7 @@ class UserTypes(object):
|
||||
ALL_USER_TYPES = (SUPPORT, BOT)
|
||||
|
||||
|
||||
class RelationTypes(object):
|
||||
class RelationTypes:
|
||||
"""The types of relations known to this server.
|
||||
"""
|
||||
|
||||
@@ -134,14 +134,14 @@ class RelationTypes(object):
|
||||
REFERENCE = "m.reference"
|
||||
|
||||
|
||||
class LimitBlockingTypes(object):
|
||||
class LimitBlockingTypes:
|
||||
"""Reasons that a server may be blocked"""
|
||||
|
||||
MONTHLY_ACTIVE_USER = "monthly_active_user"
|
||||
HS_DISABLED = "hs_disabled"
|
||||
|
||||
|
||||
class EventContentFields(object):
|
||||
class EventContentFields:
|
||||
"""Fields found in events' content, regardless of type."""
|
||||
|
||||
# Labels for the event, cf https://github.com/matrix-org/matrix-doc/pull/2326
|
||||
@@ -150,3 +150,8 @@ class EventContentFields(object):
|
||||
# Timestamp to delete the event after
|
||||
# cf https://github.com/matrix-org/matrix-doc/pull/2228
|
||||
SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
|
||||
|
||||
|
||||
class RoomEncryptionAlgorithms:
|
||||
MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"
|
||||
DEFAULT = MEGOLM_V1_AES_SHA2
|
||||
|
||||
@@ -17,19 +17,21 @@
|
||||
"""Contains exceptions and error codes."""
|
||||
|
||||
import logging
|
||||
from typing import Dict, List
|
||||
|
||||
from six import iteritems
|
||||
from six.moves import http_client
|
||||
|
||||
from canonicaljson import json
|
||||
import typing
|
||||
from http import HTTPStatus
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
from twisted.web import http
|
||||
|
||||
from synapse.util import json_decoder
|
||||
|
||||
if typing.TYPE_CHECKING:
|
||||
from synapse.types import JsonDict
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Codes(object):
|
||||
class Codes:
|
||||
UNRECOGNIZED = "M_UNRECOGNIZED"
|
||||
UNAUTHORIZED = "M_UNAUTHORIZED"
|
||||
FORBIDDEN = "M_FORBIDDEN"
|
||||
@@ -80,11 +82,11 @@ class CodeMessageException(RuntimeError):
|
||||
"""An exception with integer code and message string attributes.
|
||||
|
||||
Attributes:
|
||||
code (int): HTTP error code
|
||||
msg (str): string describing the error
|
||||
code: HTTP error code
|
||||
msg: string describing the error
|
||||
"""
|
||||
|
||||
def __init__(self, code, msg):
|
||||
def __init__(self, code: Union[int, HTTPStatus], msg: str):
|
||||
super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
|
||||
|
||||
# Some calls to this method pass instances of http.HTTPStatus for `code`.
|
||||
@@ -125,16 +127,16 @@ class SynapseError(CodeMessageException):
|
||||
message (as well as an HTTP status code).
|
||||
|
||||
Attributes:
|
||||
errcode (str): Matrix error code e.g 'M_FORBIDDEN'
|
||||
errcode: Matrix error code e.g 'M_FORBIDDEN'
|
||||
"""
|
||||
|
||||
def __init__(self, code, msg, errcode=Codes.UNKNOWN):
|
||||
def __init__(self, code: int, msg: str, errcode: str = Codes.UNKNOWN):
|
||||
"""Constructs a synapse error.
|
||||
|
||||
Args:
|
||||
code (int): The integer error code (an HTTP response code)
|
||||
msg (str): The human-readable error message.
|
||||
errcode (str): The matrix error code e.g 'M_FORBIDDEN'
|
||||
code: The integer error code (an HTTP response code)
|
||||
msg: The human-readable error message.
|
||||
errcode: The matrix error code e.g 'M_FORBIDDEN'
|
||||
"""
|
||||
super(SynapseError, self).__init__(code, msg)
|
||||
self.errcode = errcode
|
||||
@@ -147,10 +149,16 @@ class ProxiedRequestError(SynapseError):
|
||||
"""An error from a general matrix endpoint, eg. from a proxied Matrix API call.
|
||||
|
||||
Attributes:
|
||||
errcode (str): Matrix error code e.g 'M_FORBIDDEN'
|
||||
errcode: Matrix error code e.g 'M_FORBIDDEN'
|
||||
"""
|
||||
|
||||
def __init__(self, code, msg, errcode=Codes.UNKNOWN, additional_fields=None):
|
||||
def __init__(
|
||||
self,
|
||||
code: int,
|
||||
msg: str,
|
||||
errcode: str = Codes.UNKNOWN,
|
||||
additional_fields: Optional[Dict] = None,
|
||||
):
|
||||
super(ProxiedRequestError, self).__init__(code, msg, errcode)
|
||||
if additional_fields is None:
|
||||
self._additional_fields = {} # type: Dict
|
||||
@@ -166,15 +174,15 @@ class ConsentNotGivenError(SynapseError):
|
||||
privacy policy.
|
||||
"""
|
||||
|
||||
def __init__(self, msg, consent_uri):
|
||||
def __init__(self, msg: str, consent_uri: str):
|
||||
"""Constructs a ConsentNotGivenError
|
||||
|
||||
Args:
|
||||
msg (str): The human-readable error message
|
||||
consent_url (str): The URL where the user can give their consent
|
||||
msg: The human-readable error message
|
||||
consent_url: The URL where the user can give their consent
|
||||
"""
|
||||
super(ConsentNotGivenError, self).__init__(
|
||||
code=http_client.FORBIDDEN, msg=msg, errcode=Codes.CONSENT_NOT_GIVEN
|
||||
code=HTTPStatus.FORBIDDEN, msg=msg, errcode=Codes.CONSENT_NOT_GIVEN
|
||||
)
|
||||
self._consent_uri = consent_uri
|
||||
|
||||
@@ -187,14 +195,14 @@ class UserDeactivatedError(SynapseError):
|
||||
authenticated endpoint, but the account has been deactivated.
|
||||
"""
|
||||
|
||||
def __init__(self, msg):
|
||||
def __init__(self, msg: str):
|
||||
"""Constructs a UserDeactivatedError
|
||||
|
||||
Args:
|
||||
msg (str): The human-readable error message
|
||||
msg: The human-readable error message
|
||||
"""
|
||||
super(UserDeactivatedError, self).__init__(
|
||||
code=http_client.FORBIDDEN, msg=msg, errcode=Codes.USER_DEACTIVATED
|
||||
code=HTTPStatus.FORBIDDEN, msg=msg, errcode=Codes.USER_DEACTIVATED
|
||||
)
|
||||
|
||||
|
||||
@@ -203,16 +211,16 @@ class FederationDeniedError(SynapseError):
|
||||
is not on its federation whitelist.
|
||||
|
||||
Attributes:
|
||||
destination (str): The destination which has been denied
|
||||
destination: The destination which has been denied
|
||||
"""
|
||||
|
||||
def __init__(self, destination):
|
||||
def __init__(self, destination: Optional[str]):
|
||||
"""Raised by federation client or server to indicate that we are
|
||||
are deliberately not attempting to contact a given server because it is
|
||||
not on our federation whitelist.
|
||||
|
||||
Args:
|
||||
destination (str): the domain in question
|
||||
destination: the domain in question
|
||||
"""
|
||||
|
||||
self.destination = destination
|
||||
@@ -230,14 +238,16 @@ class InteractiveAuthIncompleteError(Exception):
|
||||
(This indicates we should return a 401 with 'result' as the body)
|
||||
|
||||
Attributes:
|
||||
result (dict): the server response to the request, which should be
|
||||
session_id: The ID of the ongoing interactive auth session.
|
||||
result: the server response to the request, which should be
|
||||
passed back to the client
|
||||
"""
|
||||
|
||||
def __init__(self, result):
|
||||
def __init__(self, session_id: str, result: "JsonDict"):
|
||||
super(InteractiveAuthIncompleteError, self).__init__(
|
||||
"Interactive auth not yet complete"
|
||||
)
|
||||
self.session_id = session_id
|
||||
self.result = result
|
||||
|
||||
|
||||
@@ -247,7 +257,6 @@ class UnrecognizedRequestError(SynapseError):
|
||||
def __init__(self, *args, **kwargs):
|
||||
if "errcode" not in kwargs:
|
||||
kwargs["errcode"] = Codes.UNRECOGNIZED
|
||||
message = None
|
||||
if len(args) == 0:
|
||||
message = "Unrecognized request"
|
||||
else:
|
||||
@@ -258,7 +267,7 @@ class UnrecognizedRequestError(SynapseError):
|
||||
class NotFoundError(SynapseError):
|
||||
"""An error indicating we can't find the thing you asked for"""
|
||||
|
||||
def __init__(self, msg="Not found", errcode=Codes.NOT_FOUND):
|
||||
def __init__(self, msg: str = "Not found", errcode: str = Codes.NOT_FOUND):
|
||||
super(NotFoundError, self).__init__(404, msg, errcode=errcode)
|
||||
|
||||
|
||||
@@ -284,21 +293,23 @@ class InvalidClientCredentialsError(SynapseError):
|
||||
M_UNKNOWN_TOKEN respectively.
|
||||
"""
|
||||
|
||||
def __init__(self, msg, errcode):
|
||||
def __init__(self, msg: str, errcode: str):
|
||||
super().__init__(code=401, msg=msg, errcode=errcode)
|
||||
|
||||
|
||||
class MissingClientTokenError(InvalidClientCredentialsError):
|
||||
"""Raised when we couldn't find the access token in a request"""
|
||||
|
||||
def __init__(self, msg="Missing access token"):
|
||||
def __init__(self, msg: str = "Missing access token"):
|
||||
super().__init__(msg=msg, errcode="M_MISSING_TOKEN")
|
||||
|
||||
|
||||
class InvalidClientTokenError(InvalidClientCredentialsError):
|
||||
"""Raised when we didn't understand the access token in a request"""
|
||||
|
||||
def __init__(self, msg="Unrecognised access token", soft_logout=False):
|
||||
def __init__(
|
||||
self, msg: str = "Unrecognised access token", soft_logout: bool = False
|
||||
):
|
||||
super().__init__(msg=msg, errcode="M_UNKNOWN_TOKEN")
|
||||
self._soft_logout = soft_logout
|
||||
|
||||
@@ -316,11 +327,11 @@ class ResourceLimitError(SynapseError):
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
code,
|
||||
msg,
|
||||
errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
|
||||
admin_contact=None,
|
||||
limit_type=None,
|
||||
code: int,
|
||||
msg: str,
|
||||
errcode: str = Codes.RESOURCE_LIMIT_EXCEEDED,
|
||||
admin_contact: Optional[str] = None,
|
||||
limit_type: Optional[str] = None,
|
||||
):
|
||||
self.admin_contact = admin_contact
|
||||
self.limit_type = limit_type
|
||||
@@ -368,10 +379,10 @@ class StoreError(SynapseError):
|
||||
class InvalidCaptchaError(SynapseError):
|
||||
def __init__(
|
||||
self,
|
||||
code=400,
|
||||
msg="Invalid captcha.",
|
||||
error_url=None,
|
||||
errcode=Codes.CAPTCHA_INVALID,
|
||||
code: int = 400,
|
||||
msg: str = "Invalid captcha.",
|
||||
error_url: Optional[str] = None,
|
||||
errcode: str = Codes.CAPTCHA_INVALID,
|
||||
):
|
||||
super(InvalidCaptchaError, self).__init__(code, msg, errcode)
|
||||
self.error_url = error_url
|
||||
@@ -386,10 +397,10 @@ class LimitExceededError(SynapseError):
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
code=429,
|
||||
msg="Too Many Requests",
|
||||
retry_after_ms=None,
|
||||
errcode=Codes.LIMIT_EXCEEDED,
|
||||
code: int = 429,
|
||||
msg: str = "Too Many Requests",
|
||||
retry_after_ms: Optional[int] = None,
|
||||
errcode: str = Codes.LIMIT_EXCEEDED,
|
||||
):
|
||||
super(LimitExceededError, self).__init__(code, msg, errcode)
|
||||
self.retry_after_ms = retry_after_ms
|
||||
@@ -402,10 +413,10 @@ class RoomKeysVersionError(SynapseError):
|
||||
"""A client has tried to upload to a non-current version of the room_keys store
|
||||
"""
|
||||
|
||||
def __init__(self, current_version):
|
||||
def __init__(self, current_version: str):
|
||||
"""
|
||||
Args:
|
||||
current_version (str): the current version of the store they should have used
|
||||
current_version: the current version of the store they should have used
|
||||
"""
|
||||
super(RoomKeysVersionError, self).__init__(
|
||||
403, "Wrong room_keys version", Codes.WRONG_ROOM_KEYS_VERSION
|
||||
@@ -417,7 +428,7 @@ class UnsupportedRoomVersionError(SynapseError):
|
||||
"""The client's request to create a room used a room version that the server does
|
||||
not support."""
|
||||
|
||||
def __init__(self, msg="Homeserver does not support this room version"):
|
||||
def __init__(self, msg: str = "Homeserver does not support this room version"):
|
||||
super(UnsupportedRoomVersionError, self).__init__(
|
||||
code=400, msg=msg, errcode=Codes.UNSUPPORTED_ROOM_VERSION,
|
||||
)
|
||||
@@ -439,7 +450,7 @@ class IncompatibleRoomVersionError(SynapseError):
|
||||
failing.
|
||||
"""
|
||||
|
||||
def __init__(self, room_version):
|
||||
def __init__(self, room_version: str):
|
||||
super(IncompatibleRoomVersionError, self).__init__(
|
||||
code=400,
|
||||
msg="Your homeserver does not support the features required to "
|
||||
@@ -459,8 +470,8 @@ class PasswordRefusedError(SynapseError):
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
msg="This password doesn't comply with the server's policy",
|
||||
errcode=Codes.WEAK_PASSWORD,
|
||||
msg: str = "This password doesn't comply with the server's policy",
|
||||
errcode: str = Codes.WEAK_PASSWORD,
|
||||
):
|
||||
super(PasswordRefusedError, self).__init__(
|
||||
code=400, msg=msg, errcode=errcode,
|
||||
@@ -485,19 +496,19 @@ class RequestSendFailed(RuntimeError):
|
||||
self.can_retry = can_retry
|
||||
|
||||
|
||||
def cs_error(msg, code=Codes.UNKNOWN, **kwargs):
|
||||
def cs_error(msg: str, code: str = Codes.UNKNOWN, **kwargs):
|
||||
""" Utility method for constructing an error response for client-server
|
||||
interactions.
|
||||
|
||||
Args:
|
||||
msg (str): The error message.
|
||||
code (str): The error code.
|
||||
kwargs : Additional keys to add to the response.
|
||||
msg: The error message.
|
||||
code: The error code.
|
||||
kwargs: Additional keys to add to the response.
|
||||
Returns:
|
||||
A dict representing the error response JSON.
|
||||
"""
|
||||
err = {"error": msg, "errcode": code}
|
||||
for key, value in iteritems(kwargs):
|
||||
for key, value in kwargs.items():
|
||||
err[key] = value
|
||||
return err
|
||||
|
||||
@@ -514,7 +525,14 @@ class FederationError(RuntimeError):
|
||||
is wrong (e.g., it referred to an invalid event)
|
||||
"""
|
||||
|
||||
def __init__(self, level, code, reason, affected, source=None):
|
||||
def __init__(
|
||||
self,
|
||||
level: str,
|
||||
code: int,
|
||||
reason: str,
|
||||
affected: str,
|
||||
source: Optional[str] = None,
|
||||
):
|
||||
if level not in ["FATAL", "ERROR", "WARN"]:
|
||||
raise ValueError("Level is not valid: %s" % (level,))
|
||||
self.level = level
|
||||
@@ -541,16 +559,16 @@ class HttpResponseException(CodeMessageException):
|
||||
Represents an HTTP-level failure of an outbound request
|
||||
|
||||
Attributes:
|
||||
response (bytes): body of response
|
||||
response: body of response
|
||||
"""
|
||||
|
||||
def __init__(self, code, msg, response):
|
||||
def __init__(self, code: int, msg: str, response: bytes):
|
||||
"""
|
||||
|
||||
Args:
|
||||
code (int): HTTP status code
|
||||
msg (str): reason phrase from HTTP response status line
|
||||
response (bytes): body of response
|
||||
code: HTTP status code
|
||||
msg: reason phrase from HTTP response status line
|
||||
response: body of response
|
||||
"""
|
||||
super(HttpResponseException, self).__init__(code, msg)
|
||||
self.response = response
|
||||
@@ -575,7 +593,7 @@ class HttpResponseException(CodeMessageException):
|
||||
# try to parse the body as json, to get better errcode/msg, but
|
||||
# default to M_UNKNOWN with the HTTP status as the error text
|
||||
try:
|
||||
j = json.loads(self.response)
|
||||
j = json_decoder.decode(self.response.decode("utf-8"))
|
||||
except ValueError:
|
||||
j = {}
|
||||
|
||||
@@ -586,3 +604,11 @@ class HttpResponseException(CodeMessageException):
|
||||
errmsg = j.pop("error", self.msg)
|
||||
|
||||
return ProxiedRequestError(self.code, errmsg, errcode, j)
|
||||
|
||||
|
||||
class ShadowBanError(Exception):
|
||||
"""
|
||||
Raised when a shadow-banned user attempts to perform an action.
|
||||
|
||||
This should be caught and a proper "fake" success response sent to the user.
|
||||
"""
|
||||
|
||||
@@ -17,17 +17,13 @@
|
||||
# limitations under the License.
|
||||
from typing import List
|
||||
|
||||
from six import text_type
|
||||
|
||||
import jsonschema
|
||||
from canonicaljson import json
|
||||
from jsonschema import FormatChecker
|
||||
|
||||
from twisted.internet import defer
|
||||
|
||||
from synapse.api.constants import EventContentFields
|
||||
from synapse.api.errors import SynapseError
|
||||
from synapse.storage.presence import UserPresenceState
|
||||
from synapse.api.presence import UserPresenceState
|
||||
from synapse.types import RoomID, UserID
|
||||
|
||||
FILTER_SCHEMA = {
|
||||
@@ -134,14 +130,13 @@ def matrix_user_id_validator(user_id_str):
|
||||
return UserID.from_string(user_id_str)
|
||||
|
||||
|
||||
class Filtering(object):
|
||||
class Filtering:
|
||||
def __init__(self, hs):
|
||||
super(Filtering, self).__init__()
|
||||
self.store = hs.get_datastore()
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def get_user_filter(self, user_localpart, filter_id):
|
||||
result = yield self.store.get_user_filter(user_localpart, filter_id)
|
||||
async def get_user_filter(self, user_localpart, filter_id):
|
||||
result = await self.store.get_user_filter(user_localpart, filter_id)
|
||||
return FilterCollection(result)
|
||||
|
||||
def add_user_filter(self, user_localpart, user_filter):
|
||||
@@ -173,7 +168,7 @@ class Filtering(object):
|
||||
raise SynapseError(400, str(e))
|
||||
|
||||
|
||||
class FilterCollection(object):
|
||||
class FilterCollection:
|
||||
def __init__(self, filter_json):
|
||||
self._filter_json = filter_json
|
||||
|
||||
@@ -254,7 +249,7 @@ class FilterCollection(object):
|
||||
)
|
||||
|
||||
|
||||
class Filter(object):
|
||||
class Filter:
|
||||
def __init__(self, filter_json):
|
||||
self.filter_json = filter_json
|
||||
|
||||
@@ -313,7 +308,7 @@ class Filter(object):
|
||||
|
||||
content = event.get("content", {})
|
||||
# check if there is a string url field in the content for filtering purposes
|
||||
contains_url = isinstance(content.get("url"), text_type)
|
||||
contains_url = isinstance(content.get("url"), str)
|
||||
labels = content.get(EventContentFields.LABELS, [])
|
||||
|
||||
return self.check_fields(room_id, sender, ev_type, labels, contains_url)
|
||||
|
||||
@@ -17,10 +17,11 @@ from collections import OrderedDict
|
||||
from typing import Any, Optional, Tuple
|
||||
|
||||
from synapse.api.errors import LimitExceededError
|
||||
from synapse.types import Requester
|
||||
from synapse.util import Clock
|
||||
|
||||
|
||||
class Ratelimiter(object):
|
||||
class Ratelimiter:
|
||||
"""
|
||||
Ratelimit actions marked by arbitrary keys.
|
||||
|
||||
@@ -43,6 +44,42 @@ class Ratelimiter(object):
|
||||
# * The rate_hz of this particular entry. This can vary per request
|
||||
self.actions = OrderedDict() # type: OrderedDict[Any, Tuple[float, int, float]]
|
||||
|
||||
def can_requester_do_action(
|
||||
self,
|
||||
requester: Requester,
|
||||
rate_hz: Optional[float] = None,
|
||||
burst_count: Optional[int] = None,
|
||||
update: bool = True,
|
||||
_time_now_s: Optional[int] = None,
|
||||
) -> Tuple[bool, float]:
|
||||
"""Can the requester perform the action?
|
||||
|
||||
Args:
|
||||
requester: The requester to key off when rate limiting. The user property
|
||||
will be used.
|
||||
rate_hz: The long term number of actions that can be performed in a second.
|
||||
Overrides the value set during instantiation if set.
|
||||
burst_count: How many actions that can be performed before being limited.
|
||||
Overrides the value set during instantiation if set.
|
||||
update: Whether to count this check as performing the action
|
||||
_time_now_s: The current time. Optional, defaults to the current time according
|
||||
to self.clock. Only used by tests.
|
||||
|
||||
Returns:
|
||||
A tuple containing:
|
||||
* A bool indicating if they can perform the action now
|
||||
* The reactor timestamp for when the action can be performed next.
|
||||
-1 if rate_hz is less than or equal to zero
|
||||
"""
|
||||
# Disable rate limiting of users belonging to any AS that is configured
|
||||
# not to be rate limited in its registration file (rate_limited: true|false).
|
||||
if requester.app_service and not requester.app_service.is_rate_limited():
|
||||
return True, -1.0
|
||||
|
||||
return self.can_do_action(
|
||||
requester.user.to_string(), rate_hz, burst_count, update, _time_now_s
|
||||
)
|
||||
|
||||
def can_do_action(
|
||||
self,
|
||||
key: Any,
|
||||
|
||||
@@ -18,7 +18,7 @@ from typing import Dict
|
||||
import attr
|
||||
|
||||
|
||||
class EventFormatVersions(object):
|
||||
class EventFormatVersions:
|
||||
"""This is an internal enum for tracking the version of the event format,
|
||||
independently from the room version.
|
||||
"""
|
||||
@@ -35,20 +35,20 @@ KNOWN_EVENT_FORMAT_VERSIONS = {
|
||||
}
|
||||
|
||||
|
||||
class StateResolutionVersions(object):
|
||||
class StateResolutionVersions:
|
||||
"""Enum to identify the state resolution algorithms"""
|
||||
|
||||
V1 = 1 # room v1 state res
|
||||
V2 = 2 # MSC1442 state res: room v2 and later
|
||||
|
||||
|
||||
class RoomDisposition(object):
|
||||
class RoomDisposition:
|
||||
STABLE = "stable"
|
||||
UNSTABLE = "unstable"
|
||||
|
||||
|
||||
@attr.s(slots=True, frozen=True)
|
||||
class RoomVersion(object):
|
||||
class RoomVersion:
|
||||
"""An object which describes the unique attributes of a room version."""
|
||||
|
||||
identifier = attr.ib() # str; the identifier for this version
|
||||
@@ -69,7 +69,7 @@ class RoomVersion(object):
|
||||
limit_notifications_power_levels = attr.ib(type=bool)
|
||||
|
||||
|
||||
class RoomVersions(object):
|
||||
class RoomVersions:
|
||||
V1 = RoomVersion(
|
||||
"1",
|
||||
RoomDisposition.STABLE,
|
||||
|
||||
@@ -17,8 +17,7 @@
|
||||
"""Contains the URL paths to prefix various aspects of the server with. """
|
||||
import hmac
|
||||
from hashlib import sha256
|
||||
|
||||
from six.moves.urllib.parse import urlencode
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from synapse.config import ConfigError
|
||||
|
||||
@@ -34,7 +33,7 @@ MEDIA_PREFIX = "/_matrix/media/r0"
|
||||
LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
|
||||
|
||||
|
||||
class ConsentURIBuilder(object):
|
||||
class ConsentURIBuilder:
|
||||
def __init__(self, hs_config):
|
||||
"""
|
||||
Args:
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import gc
|
||||
import logging
|
||||
import os
|
||||
@@ -20,8 +19,8 @@ import signal
|
||||
import socket
|
||||
import sys
|
||||
import traceback
|
||||
from typing import Iterable
|
||||
|
||||
from daemonize import Daemonize
|
||||
from typing_extensions import NoReturn
|
||||
|
||||
from twisted.internet import defer, error, reactor
|
||||
@@ -29,9 +28,11 @@ from twisted.protocols.tls import TLSMemoryBIOFactory
|
||||
|
||||
import synapse
|
||||
from synapse.app import check_bind_error
|
||||
from synapse.config.server import ListenerConfig
|
||||
from synapse.crypto import context_factory
|
||||
from synapse.logging.context import PreserveLoggingContext
|
||||
from synapse.util.async_helpers import Linearizer
|
||||
from synapse.util.daemonize import daemonize_process
|
||||
from synapse.util.rlimit import change_resource_limit
|
||||
from synapse.util.versionstring import get_version_string
|
||||
|
||||
@@ -127,17 +128,8 @@ def start_reactor(
|
||||
if print_pidfile:
|
||||
print(pid_file)
|
||||
|
||||
daemon = Daemonize(
|
||||
app=appname,
|
||||
pid=pid_file,
|
||||
action=run,
|
||||
auto_close_fds=False,
|
||||
verbose=True,
|
||||
logger=logger,
|
||||
)
|
||||
daemon.start()
|
||||
else:
|
||||
run()
|
||||
daemonize_process(pid_file, logger)
|
||||
run()
|
||||
|
||||
|
||||
def quit_with_error(error_string: str) -> NoReturn:
|
||||
@@ -234,7 +226,7 @@ def refresh_certificate(hs):
|
||||
logger.info("Context factories updated.")
|
||||
|
||||
|
||||
def start(hs, listeners=None):
|
||||
def start(hs: "synapse.server.HomeServer", listeners: Iterable[ListenerConfig]):
|
||||
"""
|
||||
Start a Synapse server or worker.
|
||||
|
||||
@@ -245,8 +237,8 @@ def start(hs, listeners=None):
|
||||
notify systemd.
|
||||
|
||||
Args:
|
||||
hs (synapse.server.HomeServer)
|
||||
listeners (list[dict]): Listener configuration ('listeners' in homeserver.yaml)
|
||||
hs: homeserver instance
|
||||
listeners: Listener configuration ('listeners' in homeserver.yaml)
|
||||
"""
|
||||
try:
|
||||
# Set up the SIGHUP machinery.
|
||||
@@ -276,7 +268,7 @@ def start(hs, listeners=None):
|
||||
|
||||
# It is now safe to start your Synapse.
|
||||
hs.start_listening(listeners)
|
||||
hs.get_datastore().db.start_profiling()
|
||||
hs.get_datastore().db_pool.start_profiling()
|
||||
hs.get_pusherpool().start()
|
||||
|
||||
setup_sentry(hs)
|
||||
@@ -342,6 +334,13 @@ def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
|
||||
This is to workaround https://twistedmatrix.com/trac/ticket/9620, where we
|
||||
can run out of file descriptors and infinite loop if we attempt to do too
|
||||
many DNS queries at once
|
||||
|
||||
XXX: I'm confused by this. reactor.nameResolver does not use twisted.names unless
|
||||
you explicitly install twisted.names as the resolver; rather it uses a GAIResolver
|
||||
backed by the reactor's default threadpool (which is limited to 10 threads). So
|
||||
(a) I don't understand why twisted ticket 9620 is relevant, and (b) I don't
|
||||
understand why we would run out of FDs if we did too many lookups at once.
|
||||
-- richvdh 2020/08/29
|
||||
"""
|
||||
new_resolver = _LimitedHostnameResolver(
|
||||
reactor.nameResolver, max_dns_requests_in_flight
|
||||
@@ -350,7 +349,7 @@ def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
|
||||
reactor.installNameResolver(new_resolver)
|
||||
|
||||
|
||||
class _LimitedHostnameResolver(object):
|
||||
class _LimitedHostnameResolver:
|
||||
"""Wraps a IHostnameResolver, limiting the number of in-flight DNS lookups.
|
||||
"""
|
||||
|
||||
@@ -410,7 +409,7 @@ class _LimitedHostnameResolver(object):
|
||||
yield deferred
|
||||
|
||||
|
||||
class _DeferredResolutionReceiver(object):
|
||||
class _DeferredResolutionReceiver:
|
||||
"""Wraps a IResolutionReceiver and simply resolves the given deferred when
|
||||
resolution is complete
|
||||
"""
|
||||
|
||||
@@ -79,8 +79,7 @@ class AdminCmdServer(HomeServer):
|
||||
pass
|
||||
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def export_data_command(hs, args):
|
||||
async def export_data_command(hs, args):
|
||||
"""Export data for a user.
|
||||
|
||||
Args:
|
||||
@@ -91,10 +90,8 @@ def export_data_command(hs, args):
|
||||
user_id = args.user_id
|
||||
directory = args.output_directory
|
||||
|
||||
res = yield defer.ensureDeferred(
|
||||
hs.get_handlers().admin_handler.export_user_data(
|
||||
user_id, FileExfiltrationWriter(user_id, directory=directory)
|
||||
)
|
||||
res = await hs.get_handlers().admin_handler.export_user_data(
|
||||
user_id, FileExfiltrationWriter(user_id, directory=directory)
|
||||
)
|
||||
print(res)
|
||||
|
||||
@@ -232,14 +229,15 @@ def start(config_options):
|
||||
# We also make sure that `_base.start` gets run before we actually run the
|
||||
# command.
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def run(_reactor):
|
||||
async def run():
|
||||
with LoggingContext("command"):
|
||||
yield _base.start(ss, [])
|
||||
yield args.func(ss, args)
|
||||
_base.start(ss, [])
|
||||
await args.func(ss, args)
|
||||
|
||||
_base.start_worker_reactor(
|
||||
"synapse-admin-cmd", config, run_command=lambda: task.react(run)
|
||||
"synapse-admin-cmd",
|
||||
config,
|
||||
run_command=lambda: task.react(lambda _reactor: defer.ensureDeferred(run())),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ from typing import Dict, Iterable, Optional, Set
|
||||
|
||||
from typing_extensions import ContextManager
|
||||
|
||||
from twisted.internet import defer, reactor
|
||||
from twisted.internet import address, reactor
|
||||
|
||||
import synapse
|
||||
import synapse.events
|
||||
@@ -37,6 +37,7 @@ from synapse.app import _base
|
||||
from synapse.config._base import ConfigError
|
||||
from synapse.config.homeserver import HomeServerConfig
|
||||
from synapse.config.logger import setup_logging
|
||||
from synapse.config.server import ListenerConfig
|
||||
from synapse.federation import send_queue
|
||||
from synapse.federation.transport.server import TransportLayerServer
|
||||
from synapse.handlers.presence import (
|
||||
@@ -86,7 +87,6 @@ from synapse.replication.tcp.streams import (
|
||||
ReceiptsStream,
|
||||
TagAccountDataStream,
|
||||
ToDeviceStream,
|
||||
TypingStream,
|
||||
)
|
||||
from synapse.rest.admin import register_servlets_for_media_repo
|
||||
from synapse.rest.client.v1 import events
|
||||
@@ -110,6 +110,7 @@ from synapse.rest.client.v1.room import (
|
||||
RoomSendEventRestServlet,
|
||||
RoomStateEventRestServlet,
|
||||
RoomStateRestServlet,
|
||||
RoomTypingRestServlet,
|
||||
)
|
||||
from synapse.rest.client.v1.voip import VoipRestServlet
|
||||
from synapse.rest.client.v2_alpha import groups, sync, user_directory
|
||||
@@ -122,17 +123,18 @@ from synapse.rest.client.v2_alpha.account_data import (
|
||||
from synapse.rest.client.v2_alpha.keys import KeyChangesServlet, KeyQueryServlet
|
||||
from synapse.rest.client.v2_alpha.register import RegisterRestServlet
|
||||
from synapse.rest.client.versions import VersionsRestServlet
|
||||
from synapse.rest.health import HealthResource
|
||||
from synapse.rest.key.v2 import KeyApiV2Resource
|
||||
from synapse.server import HomeServer
|
||||
from synapse.storage.data_stores.main.censor_events import CensorEventsStore
|
||||
from synapse.storage.data_stores.main.media_repository import MediaRepositoryStore
|
||||
from synapse.storage.data_stores.main.monthly_active_users import (
|
||||
from synapse.server import HomeServer, cache_in_self
|
||||
from synapse.storage.databases.main.censor_events import CensorEventsStore
|
||||
from synapse.storage.databases.main.media_repository import MediaRepositoryStore
|
||||
from synapse.storage.databases.main.monthly_active_users import (
|
||||
MonthlyActiveUsersWorkerStore,
|
||||
)
|
||||
from synapse.storage.data_stores.main.presence import UserPresenceState
|
||||
from synapse.storage.data_stores.main.search import SearchWorkerStore
|
||||
from synapse.storage.data_stores.main.ui_auth import UIAuthWorkerStore
|
||||
from synapse.storage.data_stores.main.user_directory import UserDirectoryStore
|
||||
from synapse.storage.databases.main.presence import UserPresenceState
|
||||
from synapse.storage.databases.main.search import SearchWorkerStore
|
||||
from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
|
||||
from synapse.storage.databases.main.user_directory import UserDirectoryStore
|
||||
from synapse.types import ReadReceipt
|
||||
from synapse.util.async_helpers import Linearizer
|
||||
from synapse.util.httpresourcetree import create_resource_tree
|
||||
@@ -205,10 +207,30 @@ class KeyUploadServlet(RestServlet):
|
||||
|
||||
if body:
|
||||
# They're actually trying to upload something, proxy to main synapse.
|
||||
# Pass through the auth headers, if any, in case the access token
|
||||
# is there.
|
||||
auth_headers = request.requestHeaders.getRawHeaders(b"Authorization", [])
|
||||
headers = {"Authorization": auth_headers}
|
||||
|
||||
# Proxy headers from the original request, such as the auth headers
|
||||
# (in case the access token is there) and the original IP /
|
||||
# User-Agent of the request.
|
||||
headers = {
|
||||
header: request.requestHeaders.getRawHeaders(header, [])
|
||||
for header in (b"Authorization", b"User-Agent")
|
||||
}
|
||||
# Add the previous hop the the X-Forwarded-For header.
|
||||
x_forwarded_for = request.requestHeaders.getRawHeaders(
|
||||
b"X-Forwarded-For", []
|
||||
)
|
||||
if isinstance(request.client, (address.IPv4Address, address.IPv6Address)):
|
||||
previous_host = request.client.host.encode("ascii")
|
||||
# If the header exists, add to the comma-separated list of the first
|
||||
# instance of the header. Otherwise, generate a new header.
|
||||
if x_forwarded_for:
|
||||
x_forwarded_for = [
|
||||
x_forwarded_for[0] + b", " + previous_host
|
||||
] + x_forwarded_for[1:]
|
||||
else:
|
||||
x_forwarded_for = [previous_host]
|
||||
headers[b"X-Forwarded-For"] = x_forwarded_for
|
||||
|
||||
try:
|
||||
result = await self.http_client.post_json_get_json(
|
||||
self.main_uri + request.uri.decode("ascii"), body, headers=headers
|
||||
@@ -353,9 +375,8 @@ class GenericWorkerPresence(BasePresenceHandler):
|
||||
|
||||
return _user_syncing()
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def notify_from_replication(self, states, stream_id):
|
||||
parties = yield get_interested_parties(self.store, states)
|
||||
async def notify_from_replication(self, states, stream_id):
|
||||
parties = await get_interested_parties(self.store, states)
|
||||
room_ids_to_states, users_to_states = parties
|
||||
|
||||
self.notifier.on_new_event(
|
||||
@@ -365,8 +386,7 @@ class GenericWorkerPresence(BasePresenceHandler):
|
||||
users=users_to_states.keys(),
|
||||
)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def process_replication_rows(self, token, rows):
|
||||
async def process_replication_rows(self, token, rows):
|
||||
states = [
|
||||
UserPresenceState(
|
||||
row.user_id,
|
||||
@@ -384,7 +404,7 @@ class GenericWorkerPresence(BasePresenceHandler):
|
||||
self.user_to_current_state[state.user_id] = state
|
||||
|
||||
stream_id = token
|
||||
yield self.notify_from_replication(states, stream_id)
|
||||
await self.notify_from_replication(states, stream_id)
|
||||
|
||||
def get_currently_syncing_users_for_replication(self) -> Iterable[str]:
|
||||
return [
|
||||
@@ -430,37 +450,6 @@ class GenericWorkerPresence(BasePresenceHandler):
|
||||
await self._bump_active_client(user_id=user_id)
|
||||
|
||||
|
||||
class GenericWorkerTyping(object):
|
||||
def __init__(self, hs):
|
||||
self._latest_room_serial = 0
|
||||
self._reset()
|
||||
|
||||
def _reset(self):
|
||||
"""
|
||||
Reset the typing handler's data caches.
|
||||
"""
|
||||
# map room IDs to serial numbers
|
||||
self._room_serials = {}
|
||||
# map room IDs to sets of users currently typing
|
||||
self._room_typing = {}
|
||||
|
||||
def process_replication_rows(self, token, rows):
|
||||
if self._latest_room_serial > token:
|
||||
# The master has gone backwards. To prevent inconsistent data, just
|
||||
# clear everything.
|
||||
self._reset()
|
||||
|
||||
# Set the latest serial token to whatever the server gave us.
|
||||
self._latest_room_serial = token
|
||||
|
||||
for row in rows:
|
||||
self._room_serials[row.room_id] = token
|
||||
self._room_typing[row.room_id] = row.user_ids
|
||||
|
||||
def get_current_token(self) -> int:
|
||||
return self._latest_room_serial
|
||||
|
||||
|
||||
class GenericWorkerSlavedStore(
|
||||
# FIXME(#3714): We need to add UserDirectoryStore as we write directly
|
||||
# rather than going via the correct worker.
|
||||
@@ -490,37 +479,27 @@ class GenericWorkerSlavedStore(
|
||||
SearchWorkerStore,
|
||||
BaseSlavedStore,
|
||||
):
|
||||
def __init__(self, database, db_conn, hs):
|
||||
super(GenericWorkerSlavedStore, self).__init__(database, db_conn, hs)
|
||||
|
||||
# We pull out the current federation stream position now so that we
|
||||
# always have a known value for the federation position in memory so
|
||||
# that we don't have to bounce via a deferred once when we start the
|
||||
# replication streams.
|
||||
self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
|
||||
|
||||
def _get_federation_out_pos(self, db_conn):
|
||||
sql = "SELECT stream_id FROM federation_stream_position WHERE type = ?"
|
||||
sql = self.database_engine.convert_param_style(sql)
|
||||
|
||||
txn = db_conn.cursor()
|
||||
txn.execute(sql, ("federation",))
|
||||
rows = txn.fetchall()
|
||||
txn.close()
|
||||
|
||||
return rows[0][0] if rows else -1
|
||||
pass
|
||||
|
||||
|
||||
class GenericWorkerServer(HomeServer):
|
||||
DATASTORE_CLASS = GenericWorkerSlavedStore
|
||||
|
||||
def _listen_http(self, listener_config):
|
||||
port = listener_config["port"]
|
||||
bind_addresses = listener_config["bind_addresses"]
|
||||
site_tag = listener_config.get("tag", port)
|
||||
resources = {}
|
||||
for res in listener_config["resources"]:
|
||||
for name in res["names"]:
|
||||
def _listen_http(self, listener_config: ListenerConfig):
|
||||
port = listener_config.port
|
||||
bind_addresses = listener_config.bind_addresses
|
||||
|
||||
assert listener_config.http_options is not None
|
||||
|
||||
site_tag = listener_config.http_options.tag
|
||||
if site_tag is None:
|
||||
site_tag = port
|
||||
|
||||
# We always include a health resource.
|
||||
resources = {"/health": HealthResource()}
|
||||
|
||||
for res in listener_config.http_options.resources:
|
||||
for name in res.names:
|
||||
if name == "metrics":
|
||||
resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
|
||||
elif name == "client":
|
||||
@@ -550,6 +529,7 @@ class GenericWorkerServer(HomeServer):
|
||||
KeyUploadServlet(self).register(resource)
|
||||
AccountDataServlet(self).register(resource)
|
||||
RoomAccountDataServlet(self).register(resource)
|
||||
RoomTypingRestServlet(self).register(resource)
|
||||
|
||||
sync.register_servlets(self, resource)
|
||||
events.register_servlets(self, resource)
|
||||
@@ -590,7 +570,7 @@ class GenericWorkerServer(HomeServer):
|
||||
" repository is disabled. Ignoring."
|
||||
)
|
||||
|
||||
if name == "openid" and "federation" not in res["names"]:
|
||||
if name == "openid" and "federation" not in res.names:
|
||||
# Only load the openid resource separately if federation resource
|
||||
# is not specified since federation resource includes openid
|
||||
# resource.
|
||||
@@ -625,19 +605,19 @@ class GenericWorkerServer(HomeServer):
|
||||
|
||||
logger.info("Synapse worker now listening on port %d", port)
|
||||
|
||||
def start_listening(self, listeners):
|
||||
def start_listening(self, listeners: Iterable[ListenerConfig]):
|
||||
for listener in listeners:
|
||||
if listener["type"] == "http":
|
||||
if listener.type == "http":
|
||||
self._listen_http(listener)
|
||||
elif listener["type"] == "manhole":
|
||||
elif listener.type == "manhole":
|
||||
_base.listen_tcp(
|
||||
listener["bind_addresses"],
|
||||
listener["port"],
|
||||
listener.bind_addresses,
|
||||
listener.port,
|
||||
manhole(
|
||||
username="matrix", password="rabbithole", globals={"hs": self}
|
||||
),
|
||||
)
|
||||
elif listener["type"] == "metrics":
|
||||
elif listener.type == "metrics":
|
||||
if not self.get_config().enable_metrics:
|
||||
logger.warning(
|
||||
(
|
||||
@@ -646,31 +626,29 @@ class GenericWorkerServer(HomeServer):
|
||||
)
|
||||
)
|
||||
else:
|
||||
_base.listen_metrics(listener["bind_addresses"], listener["port"])
|
||||
_base.listen_metrics(listener.bind_addresses, listener.port)
|
||||
else:
|
||||
logger.warning("Unrecognized listener type: %s", listener["type"])
|
||||
logger.warning("Unsupported listener type: %s", listener.type)
|
||||
|
||||
self.get_tcp_replication().start_replication(self)
|
||||
|
||||
def remove_pusher(self, app_id, push_key, user_id):
|
||||
async def remove_pusher(self, app_id, push_key, user_id):
|
||||
self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
|
||||
|
||||
def build_replication_data_handler(self):
|
||||
@cache_in_self
|
||||
def get_replication_data_handler(self):
|
||||
return GenericWorkerReplicationHandler(self)
|
||||
|
||||
def build_presence_handler(self):
|
||||
@cache_in_self
|
||||
def get_presence_handler(self):
|
||||
return GenericWorkerPresence(self)
|
||||
|
||||
def build_typing_handler(self):
|
||||
return GenericWorkerTyping(self)
|
||||
|
||||
|
||||
class GenericWorkerReplicationHandler(ReplicationDataHandler):
|
||||
def __init__(self, hs):
|
||||
super(GenericWorkerReplicationHandler, self).__init__(hs)
|
||||
|
||||
self.store = hs.get_datastore()
|
||||
self.typing_handler = hs.get_typing_handler()
|
||||
self.presence_handler = hs.get_presence_handler() # type: GenericWorkerPresence
|
||||
self.notifier = hs.get_notifier()
|
||||
|
||||
@@ -707,11 +685,6 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
|
||||
await self.pusher_pool.on_new_receipts(
|
||||
token, token, {row.room_id for row in rows}
|
||||
)
|
||||
elif stream_name == TypingStream.NAME:
|
||||
self.typing_handler.process_replication_rows(token, rows)
|
||||
self.notifier.on_new_event(
|
||||
"typing_key", token, rooms=[row.room_id for row in rows]
|
||||
)
|
||||
elif stream_name == ToDeviceStream.NAME:
|
||||
entities = [row.entity for row in rows if row.entity.startswith("@")]
|
||||
if entities:
|
||||
@@ -738,6 +711,11 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
|
||||
except Exception:
|
||||
logger.exception("Error processing replication")
|
||||
|
||||
async def on_position(self, stream_name: str, instance_name: str, token: int):
|
||||
await super().on_position(stream_name, instance_name, token)
|
||||
# Also call on_rdata to ensure that stream positions are properly reset.
|
||||
await self.on_rdata(stream_name, instance_name, token, [])
|
||||
|
||||
def stop_pusher(self, user_id, app_id, pushkey):
|
||||
if not self.notify_pushers:
|
||||
return
|
||||
@@ -767,7 +745,7 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
|
||||
self.send_handler.wake_destination(server)
|
||||
|
||||
|
||||
class FederationSenderHandler(object):
|
||||
class FederationSenderHandler:
|
||||
"""Processes the fedration replication stream
|
||||
|
||||
This class is only instantiate on the worker responsible for sending outbound
|
||||
@@ -781,19 +759,11 @@ class FederationSenderHandler(object):
|
||||
self.federation_sender = hs.get_federation_sender()
|
||||
self._hs = hs
|
||||
|
||||
# if the worker is restarted, we want to pick up where we left off in
|
||||
# the replication stream, so load the position from the database.
|
||||
#
|
||||
# XXX is this actually worthwhile? Whenever the master is restarted, we'll
|
||||
# drop some rows anyway (which is mostly fine because we're only dropping
|
||||
# typing and presence notifications). If the replication stream is
|
||||
# unreliable, why do we do all this hoop-jumping to store the position in the
|
||||
# database? See also https://github.com/matrix-org/synapse/issues/7535.
|
||||
#
|
||||
self.federation_position = self.store.federation_out_pos_startup
|
||||
# Stores the latest position in the federation stream we've gotten up
|
||||
# to. This is always set before we use it.
|
||||
self.federation_position = None
|
||||
|
||||
self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
|
||||
self._last_ack = self.federation_position
|
||||
|
||||
def on_start(self):
|
||||
# There may be some events that are persisted but haven't been sent,
|
||||
@@ -901,7 +871,6 @@ class FederationSenderHandler(object):
|
||||
# We ACK this token over replication so that the master can drop
|
||||
# its in memory queues
|
||||
self._hs.get_tcp_replication().send_federation_ack(current_position)
|
||||
self._last_ack = current_position
|
||||
except Exception:
|
||||
logger.exception("Error updating federation stream position")
|
||||
|
||||
@@ -929,7 +898,7 @@ def start(config_options):
|
||||
)
|
||||
|
||||
if config.worker_app == "synapse.app.appservice":
|
||||
if config.notify_appservices:
|
||||
if config.appservice.notify_appservices:
|
||||
sys.stderr.write(
|
||||
"\nThe appservices must be disabled in the main synapse process"
|
||||
"\nbefore they can be run in a separate worker."
|
||||
@@ -939,13 +908,13 @@ def start(config_options):
|
||||
sys.exit(1)
|
||||
|
||||
# Force the appservice to start since they will be disabled in the main config
|
||||
config.notify_appservices = True
|
||||
config.appservice.notify_appservices = True
|
||||
else:
|
||||
# For other worker types we force this to off.
|
||||
config.notify_appservices = False
|
||||
config.appservice.notify_appservices = False
|
||||
|
||||
if config.worker_app == "synapse.app.pusher":
|
||||
if config.start_pushers:
|
||||
if config.server.start_pushers:
|
||||
sys.stderr.write(
|
||||
"\nThe pushers must be disabled in the main synapse process"
|
||||
"\nbefore they can be run in a separate worker."
|
||||
@@ -955,13 +924,13 @@ def start(config_options):
|
||||
sys.exit(1)
|
||||
|
||||
# Force the pushers to start since they will be disabled in the main config
|
||||
config.start_pushers = True
|
||||
config.server.start_pushers = True
|
||||
else:
|
||||
# For other worker types we force this to off.
|
||||
config.start_pushers = False
|
||||
config.server.start_pushers = False
|
||||
|
||||
if config.worker_app == "synapse.app.user_dir":
|
||||
if config.update_user_directory:
|
||||
if config.server.update_user_directory:
|
||||
sys.stderr.write(
|
||||
"\nThe update_user_directory must be disabled in the main synapse process"
|
||||
"\nbefore they can be run in a separate worker."
|
||||
@@ -971,13 +940,13 @@ def start(config_options):
|
||||
sys.exit(1)
|
||||
|
||||
# Force the pushers to start since they will be disabled in the main config
|
||||
config.update_user_directory = True
|
||||
config.server.update_user_directory = True
|
||||
else:
|
||||
# For other worker types we force this to off.
|
||||
config.update_user_directory = False
|
||||
config.server.update_user_directory = False
|
||||
|
||||
if config.worker_app == "synapse.app.federation_sender":
|
||||
if config.send_federation:
|
||||
if config.worker.send_federation:
|
||||
sys.stderr.write(
|
||||
"\nThe send_federation must be disabled in the main synapse process"
|
||||
"\nbefore they can be run in a separate worker."
|
||||
@@ -987,10 +956,10 @@ def start(config_options):
|
||||
sys.exit(1)
|
||||
|
||||
# Force the pushers to start since they will be disabled in the main config
|
||||
config.send_federation = True
|
||||
config.worker.send_federation = True
|
||||
else:
|
||||
# For other worker types we force this to off.
|
||||
config.send_federation = False
|
||||
config.worker.send_federation = False
|
||||
|
||||
synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
|
||||
|
||||
|
||||
@@ -23,8 +23,7 @@ import math
|
||||
import os
|
||||
import resource
|
||||
import sys
|
||||
|
||||
from six import iteritems
|
||||
from typing import Iterable
|
||||
|
||||
from prometheus_client import Gauge
|
||||
|
||||
@@ -50,12 +49,14 @@ from synapse.app import _base
|
||||
from synapse.app._base import listen_ssl, listen_tcp, quit_with_error
|
||||
from synapse.config._base import ConfigError
|
||||
from synapse.config.homeserver import HomeServerConfig
|
||||
from synapse.config.server import ListenerConfig
|
||||
from synapse.federation.transport.server import TransportLayerServer
|
||||
from synapse.http.additional_resource import AdditionalResource
|
||||
from synapse.http.server import (
|
||||
OptionsResource,
|
||||
RootOptionsRedirectResource,
|
||||
RootRedirect,
|
||||
StaticResource,
|
||||
)
|
||||
from synapse.http.site import SynapseSite
|
||||
from synapse.logging.context import LoggingContext
|
||||
@@ -67,6 +68,7 @@ from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
|
||||
from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
|
||||
from synapse.rest import ClientRestResource
|
||||
from synapse.rest.admin import AdminRestResource
|
||||
from synapse.rest.health import HealthResource
|
||||
from synapse.rest.key.v2 import KeyApiV2Resource
|
||||
from synapse.rest.well_known import WellKnownResource
|
||||
from synapse.server import HomeServer
|
||||
@@ -89,24 +91,26 @@ def gz_wrap(r):
|
||||
class SynapseHomeServer(HomeServer):
|
||||
DATASTORE_CLASS = DataStore
|
||||
|
||||
def _listener_http(self, config, listener_config):
|
||||
port = listener_config["port"]
|
||||
bind_addresses = listener_config["bind_addresses"]
|
||||
tls = listener_config.get("tls", False)
|
||||
site_tag = listener_config.get("tag", port)
|
||||
def _listener_http(self, config: HomeServerConfig, listener_config: ListenerConfig):
|
||||
port = listener_config.port
|
||||
bind_addresses = listener_config.bind_addresses
|
||||
tls = listener_config.tls
|
||||
site_tag = listener_config.http_options.tag
|
||||
if site_tag is None:
|
||||
site_tag = port
|
||||
|
||||
resources = {}
|
||||
for res in listener_config["resources"]:
|
||||
for name in res["names"]:
|
||||
if name == "openid" and "federation" in res["names"]:
|
||||
# We always include a health resource.
|
||||
resources = {"/health": HealthResource()}
|
||||
|
||||
for res in listener_config.http_options.resources:
|
||||
for name in res.names:
|
||||
if name == "openid" and "federation" in res.names:
|
||||
# Skip loading openid resource if federation is defined
|
||||
# since federation resource will include openid
|
||||
continue
|
||||
resources.update(
|
||||
self._configure_named_resource(name, res.get("compress", False))
|
||||
)
|
||||
resources.update(self._configure_named_resource(name, res.compress))
|
||||
|
||||
additional_resources = listener_config.get("additional_resources", {})
|
||||
additional_resources = listener_config.http_options.additional_resources
|
||||
logger.debug("Configuring additional resources: %r", additional_resources)
|
||||
module_api = ModuleApi(self, self.get_auth_handler())
|
||||
for path, resmodule in additional_resources.items():
|
||||
@@ -228,7 +232,7 @@ class SynapseHomeServer(HomeServer):
|
||||
if name in ["static", "client"]:
|
||||
resources.update(
|
||||
{
|
||||
STATIC_PREFIX: File(
|
||||
STATIC_PREFIX: StaticResource(
|
||||
os.path.join(os.path.dirname(synapse.__file__), "static")
|
||||
)
|
||||
}
|
||||
@@ -278,7 +282,7 @@ class SynapseHomeServer(HomeServer):
|
||||
|
||||
return resources
|
||||
|
||||
def start_listening(self, listeners):
|
||||
def start_listening(self, listeners: Iterable[ListenerConfig]):
|
||||
config = self.get_config()
|
||||
|
||||
if config.redis_enabled:
|
||||
@@ -288,25 +292,25 @@ class SynapseHomeServer(HomeServer):
|
||||
self.get_tcp_replication().start_replication(self)
|
||||
|
||||
for listener in listeners:
|
||||
if listener["type"] == "http":
|
||||
if listener.type == "http":
|
||||
self._listening_services.extend(self._listener_http(config, listener))
|
||||
elif listener["type"] == "manhole":
|
||||
elif listener.type == "manhole":
|
||||
listen_tcp(
|
||||
listener["bind_addresses"],
|
||||
listener["port"],
|
||||
listener.bind_addresses,
|
||||
listener.port,
|
||||
manhole(
|
||||
username="matrix", password="rabbithole", globals={"hs": self}
|
||||
),
|
||||
)
|
||||
elif listener["type"] == "replication":
|
||||
elif listener.type == "replication":
|
||||
services = listen_tcp(
|
||||
listener["bind_addresses"],
|
||||
listener["port"],
|
||||
listener.bind_addresses,
|
||||
listener.port,
|
||||
ReplicationStreamProtocolFactory(self),
|
||||
)
|
||||
for s in services:
|
||||
reactor.addSystemEventTrigger("before", "shutdown", s.stopListening)
|
||||
elif listener["type"] == "metrics":
|
||||
elif listener.type == "metrics":
|
||||
if not self.get_config().enable_metrics:
|
||||
logger.warning(
|
||||
(
|
||||
@@ -315,9 +319,11 @@ class SynapseHomeServer(HomeServer):
|
||||
)
|
||||
)
|
||||
else:
|
||||
_base.listen_metrics(listener["bind_addresses"], listener["port"])
|
||||
_base.listen_metrics(listener.bind_addresses, listener.port)
|
||||
else:
|
||||
logger.warning("Unrecognized listener type: %s", listener["type"])
|
||||
# this shouldn't happen, as the listener type should have been checked
|
||||
# during parsing
|
||||
logger.warning("Unrecognized listener type: %s", listener.type)
|
||||
|
||||
|
||||
# Gauges to expose monthly active user control metrics
|
||||
@@ -377,13 +383,12 @@ def setup(config_options):
|
||||
|
||||
hs.setup_master()
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def do_acme():
|
||||
async def do_acme() -> bool:
|
||||
"""
|
||||
Reprovision an ACME certificate, if it's required.
|
||||
|
||||
Returns:
|
||||
Deferred[bool]: Whether the cert has been updated.
|
||||
Whether the cert has been updated.
|
||||
"""
|
||||
acme = hs.get_acme_handler()
|
||||
|
||||
@@ -402,30 +407,28 @@ def setup(config_options):
|
||||
provision = True
|
||||
|
||||
if provision:
|
||||
yield acme.provision_certificate()
|
||||
await acme.provision_certificate()
|
||||
|
||||
return provision
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def reprovision_acme():
|
||||
async def reprovision_acme():
|
||||
"""
|
||||
Provision a certificate from ACME, if required, and reload the TLS
|
||||
certificate if it's renewed.
|
||||
"""
|
||||
reprovisioned = yield do_acme()
|
||||
reprovisioned = await do_acme()
|
||||
if reprovisioned:
|
||||
_base.refresh_certificate(hs)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def start():
|
||||
async def start():
|
||||
try:
|
||||
# Run the ACME provisioning code, if it's enabled.
|
||||
if hs.config.acme_enabled:
|
||||
acme = hs.get_acme_handler()
|
||||
# Start up the webservices which we will respond to ACME
|
||||
# challenges with, and then provision.
|
||||
yield acme.start_listening()
|
||||
yield do_acme()
|
||||
await acme.start_listening()
|
||||
await do_acme()
|
||||
|
||||
# Check if it needs to be reprovisioned every day.
|
||||
hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)
|
||||
@@ -434,12 +437,12 @@ def setup(config_options):
|
||||
if hs.config.oidc_enabled:
|
||||
oidc = hs.get_oidc_handler()
|
||||
# Loading the provider metadata also ensures the provider config is valid.
|
||||
yield defer.ensureDeferred(oidc.load_metadata())
|
||||
yield defer.ensureDeferred(oidc.load_jwks())
|
||||
await oidc.load_metadata()
|
||||
await oidc.load_jwks()
|
||||
|
||||
_base.start(hs, config.listeners)
|
||||
|
||||
hs.get_datastore().db.updates.start_doing_background_updates()
|
||||
hs.get_datastore().db_pool.updates.start_doing_background_updates()
|
||||
except Exception:
|
||||
# Print the exception and bail out.
|
||||
print("Error during startup:", file=sys.stderr)
|
||||
@@ -451,7 +454,7 @@ def setup(config_options):
|
||||
reactor.stop()
|
||||
sys.exit(1)
|
||||
|
||||
reactor.callWhenRunning(start)
|
||||
reactor.callWhenRunning(lambda: defer.ensureDeferred(start()))
|
||||
|
||||
return hs
|
||||
|
||||
@@ -480,8 +483,7 @@ class SynapseService(service.Service):
|
||||
_stats_process = []
|
||||
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def phone_stats_home(hs, stats, stats_process=_stats_process):
|
||||
async def phone_stats_home(hs, stats, stats_process=_stats_process):
|
||||
logger.info("Gathering stats for reporting")
|
||||
now = int(hs.get_clock().time())
|
||||
uptime = int(now - hs.start_time)
|
||||
@@ -519,28 +521,28 @@ def phone_stats_home(hs, stats, stats_process=_stats_process):
|
||||
stats["python_version"] = "{}.{}.{}".format(
|
||||
version.major, version.minor, version.micro
|
||||
)
|
||||
stats["total_users"] = yield hs.get_datastore().count_all_users()
|
||||
stats["total_users"] = await hs.get_datastore().count_all_users()
|
||||
|
||||
total_nonbridged_users = yield hs.get_datastore().count_nonbridged_users()
|
||||
total_nonbridged_users = await hs.get_datastore().count_nonbridged_users()
|
||||
stats["total_nonbridged_users"] = total_nonbridged_users
|
||||
|
||||
daily_user_type_results = yield hs.get_datastore().count_daily_user_type()
|
||||
for name, count in iteritems(daily_user_type_results):
|
||||
daily_user_type_results = await hs.get_datastore().count_daily_user_type()
|
||||
for name, count in daily_user_type_results.items():
|
||||
stats["daily_user_type_" + name] = count
|
||||
|
||||
room_count = yield hs.get_datastore().get_room_count()
|
||||
room_count = await hs.get_datastore().get_room_count()
|
||||
stats["total_room_count"] = room_count
|
||||
|
||||
stats["daily_active_users"] = yield hs.get_datastore().count_daily_users()
|
||||
stats["monthly_active_users"] = yield hs.get_datastore().count_monthly_users()
|
||||
stats["daily_active_rooms"] = yield hs.get_datastore().count_daily_active_rooms()
|
||||
stats["daily_messages"] = yield hs.get_datastore().count_daily_messages()
|
||||
stats["daily_active_users"] = await hs.get_datastore().count_daily_users()
|
||||
stats["monthly_active_users"] = await hs.get_datastore().count_monthly_users()
|
||||
stats["daily_active_rooms"] = await hs.get_datastore().count_daily_active_rooms()
|
||||
stats["daily_messages"] = await hs.get_datastore().count_daily_messages()
|
||||
|
||||
r30_results = yield hs.get_datastore().count_r30_users()
|
||||
for name, count in iteritems(r30_results):
|
||||
r30_results = await hs.get_datastore().count_r30_users()
|
||||
for name, count in r30_results.items():
|
||||
stats["r30_users_" + name] = count
|
||||
|
||||
daily_sent_messages = yield hs.get_datastore().count_daily_sent_messages()
|
||||
daily_sent_messages = await hs.get_datastore().count_daily_sent_messages()
|
||||
stats["daily_sent_messages"] = daily_sent_messages
|
||||
stats["cache_factor"] = hs.config.caches.global_factor
|
||||
stats["event_cache_size"] = hs.config.caches.event_cache_size
|
||||
@@ -550,12 +552,12 @@ def phone_stats_home(hs, stats, stats_process=_stats_process):
|
||||
#
|
||||
|
||||
# This only reports info about the *main* database.
|
||||
stats["database_engine"] = hs.get_datastore().db.engine.module.__name__
|
||||
stats["database_server_version"] = hs.get_datastore().db.engine.server_version
|
||||
stats["database_engine"] = hs.get_datastore().db_pool.engine.module.__name__
|
||||
stats["database_server_version"] = hs.get_datastore().db_pool.engine.server_version
|
||||
|
||||
logger.info("Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats))
|
||||
try:
|
||||
yield hs.get_proxied_http_client().put_json(
|
||||
await hs.get_proxied_http_client().put_json(
|
||||
hs.config.report_stats_endpoint, stats
|
||||
)
|
||||
except Exception as e:
|
||||
|
||||
@@ -14,24 +14,25 @@
|
||||
# limitations under the License.
|
||||
import logging
|
||||
import re
|
||||
|
||||
from six import string_types
|
||||
|
||||
from twisted.internet import defer
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from synapse.api.constants import EventTypes
|
||||
from synapse.appservice.api import ApplicationServiceApi
|
||||
from synapse.types import GroupID, get_domain_from_id
|
||||
from synapse.util.caches.descriptors import cachedInlineCallbacks
|
||||
from synapse.util.caches.descriptors import cached
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from synapse.storage.databases.main import DataStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ApplicationServiceState(object):
|
||||
class ApplicationServiceState:
|
||||
DOWN = "down"
|
||||
UP = "up"
|
||||
|
||||
|
||||
class AppServiceTransaction(object):
|
||||
class AppServiceTransaction:
|
||||
"""Represents an application service transaction."""
|
||||
|
||||
def __init__(self, service, id, events):
|
||||
@@ -39,19 +40,19 @@ class AppServiceTransaction(object):
|
||||
self.id = id
|
||||
self.events = events
|
||||
|
||||
def send(self, as_api):
|
||||
async def send(self, as_api: ApplicationServiceApi) -> bool:
|
||||
"""Sends this transaction using the provided AS API interface.
|
||||
|
||||
Args:
|
||||
as_api(ApplicationServiceApi): The API to use to send.
|
||||
as_api: The API to use to send.
|
||||
Returns:
|
||||
A Deferred which resolves to True if the transaction was sent.
|
||||
True if the transaction was sent.
|
||||
"""
|
||||
return as_api.push_bulk(
|
||||
return await as_api.push_bulk(
|
||||
service=self.service, events=self.events, txn_id=self.id
|
||||
)
|
||||
|
||||
def complete(self, store):
|
||||
async def complete(self, store: "DataStore") -> None:
|
||||
"""Completes this transaction as successful.
|
||||
|
||||
Marks this transaction ID on the application service and removes the
|
||||
@@ -59,13 +60,11 @@ class AppServiceTransaction(object):
|
||||
|
||||
Args:
|
||||
store: The database store to operate on.
|
||||
Returns:
|
||||
A Deferred which resolves to True if the transaction was completed.
|
||||
"""
|
||||
return store.complete_appservice_txn(service=self.service, txn_id=self.id)
|
||||
await store.complete_appservice_txn(service=self.service, txn_id=self.id)
|
||||
|
||||
|
||||
class ApplicationService(object):
|
||||
class ApplicationService:
|
||||
"""Defines an application service. This definition is mostly what is
|
||||
provided to the /register AS API.
|
||||
|
||||
@@ -156,7 +155,7 @@ class ApplicationService(object):
|
||||
)
|
||||
|
||||
regex = regex_obj.get("regex")
|
||||
if isinstance(regex, string_types):
|
||||
if isinstance(regex, str):
|
||||
regex_obj["regex"] = re.compile(regex) # Pre-compile regex
|
||||
else:
|
||||
raise ValueError("Expected string for 'regex' in ns '%s'" % ns)
|
||||
@@ -174,8 +173,7 @@ class ApplicationService(object):
|
||||
return regex_obj["exclusive"]
|
||||
return False
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _matches_user(self, event, store):
|
||||
async def _matches_user(self, event, store):
|
||||
if not event:
|
||||
return False
|
||||
|
||||
@@ -190,12 +188,12 @@ class ApplicationService(object):
|
||||
if not store:
|
||||
return False
|
||||
|
||||
does_match = yield self._matches_user_in_member_list(event.room_id, store)
|
||||
does_match = await self._matches_user_in_member_list(event.room_id, store)
|
||||
return does_match
|
||||
|
||||
@cachedInlineCallbacks(num_args=1, cache_context=True)
|
||||
def _matches_user_in_member_list(self, room_id, store, cache_context):
|
||||
member_list = yield store.get_users_in_room(
|
||||
@cached(num_args=1, cache_context=True)
|
||||
async def _matches_user_in_member_list(self, room_id, store, cache_context):
|
||||
member_list = await store.get_users_in_room(
|
||||
room_id, on_invalidate=cache_context.invalidate
|
||||
)
|
||||
|
||||
@@ -210,35 +208,33 @@ class ApplicationService(object):
|
||||
return self.is_interested_in_room(event.room_id)
|
||||
return False
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _matches_aliases(self, event, store):
|
||||
async def _matches_aliases(self, event, store):
|
||||
if not store or not event:
|
||||
return False
|
||||
|
||||
alias_list = yield store.get_aliases_for_room(event.room_id)
|
||||
alias_list = await store.get_aliases_for_room(event.room_id)
|
||||
for alias in alias_list:
|
||||
if self.is_interested_in_alias(alias):
|
||||
return True
|
||||
return False
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def is_interested(self, event, store=None):
|
||||
async def is_interested(self, event, store=None) -> bool:
|
||||
"""Check if this service is interested in this event.
|
||||
|
||||
Args:
|
||||
event(Event): The event to check.
|
||||
store(DataStore)
|
||||
Returns:
|
||||
bool: True if this service would like to know about this event.
|
||||
True if this service would like to know about this event.
|
||||
"""
|
||||
# Do cheap checks first
|
||||
if self._matches_room_id(event):
|
||||
return True
|
||||
|
||||
if (yield self._matches_aliases(event, store)):
|
||||
if await self._matches_aliases(event, store):
|
||||
return True
|
||||
|
||||
if (yield self._matches_user(event, store)):
|
||||
if await self._matches_user(event, store):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
@@ -13,20 +13,21 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import logging
|
||||
|
||||
from six.moves import urllib
|
||||
import urllib
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
|
||||
from prometheus_client import Counter
|
||||
|
||||
from twisted.internet import defer
|
||||
|
||||
from synapse.api.constants import ThirdPartyEntityKind
|
||||
from synapse.api.constants import EventTypes, ThirdPartyEntityKind
|
||||
from synapse.api.errors import CodeMessageException
|
||||
from synapse.events.utils import serialize_event
|
||||
from synapse.http.client import SimpleHttpClient
|
||||
from synapse.types import ThirdPartyInstanceID
|
||||
from synapse.types import JsonDict, ThirdPartyInstanceID
|
||||
from synapse.util.caches.response_cache import ResponseCache
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from synapse.appservice import ApplicationService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
sent_transactions_counter = Counter(
|
||||
@@ -94,14 +95,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
hs, "as_protocol_meta", timeout_ms=HOUR_IN_MS
|
||||
)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def query_user(self, service, user_id):
|
||||
async def query_user(self, service, user_id):
|
||||
if service.url is None:
|
||||
return False
|
||||
uri = service.url + ("/users/%s" % urllib.parse.quote(user_id))
|
||||
response = None
|
||||
try:
|
||||
response = yield self.get_json(uri, {"access_token": service.hs_token})
|
||||
response = await self.get_json(uri, {"access_token": service.hs_token})
|
||||
if response is not None: # just an empty json object
|
||||
return True
|
||||
except CodeMessageException as e:
|
||||
@@ -112,14 +111,12 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
logger.warning("query_user to %s threw exception %s", uri, ex)
|
||||
return False
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def query_alias(self, service, alias):
|
||||
async def query_alias(self, service, alias):
|
||||
if service.url is None:
|
||||
return False
|
||||
uri = service.url + ("/rooms/%s" % urllib.parse.quote(alias))
|
||||
response = None
|
||||
try:
|
||||
response = yield self.get_json(uri, {"access_token": service.hs_token})
|
||||
response = await self.get_json(uri, {"access_token": service.hs_token})
|
||||
if response is not None: # just an empty json object
|
||||
return True
|
||||
except CodeMessageException as e:
|
||||
@@ -130,8 +127,7 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
logger.warning("query_alias to %s threw exception %s", uri, ex)
|
||||
return False
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def query_3pe(self, service, kind, protocol, fields):
|
||||
async def query_3pe(self, service, kind, protocol, fields):
|
||||
if kind == ThirdPartyEntityKind.USER:
|
||||
required_field = "userid"
|
||||
elif kind == ThirdPartyEntityKind.LOCATION:
|
||||
@@ -148,7 +144,7 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
urllib.parse.quote(protocol),
|
||||
)
|
||||
try:
|
||||
response = yield self.get_json(uri, fields)
|
||||
response = await self.get_json(uri, fields)
|
||||
if not isinstance(response, list):
|
||||
logger.warning(
|
||||
"query_3pe to %s returned an invalid response %r", uri, response
|
||||
@@ -169,19 +165,20 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
logger.warning("query_3pe to %s threw exception %s", uri, ex)
|
||||
return []
|
||||
|
||||
def get_3pe_protocol(self, service, protocol):
|
||||
async def get_3pe_protocol(
|
||||
self, service: "ApplicationService", protocol: str
|
||||
) -> Optional[JsonDict]:
|
||||
if service.url is None:
|
||||
return {}
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _get():
|
||||
async def _get() -> Optional[JsonDict]:
|
||||
uri = "%s%s/thirdparty/protocol/%s" % (
|
||||
service.url,
|
||||
APP_SERVICE_PREFIX,
|
||||
urllib.parse.quote(protocol),
|
||||
)
|
||||
try:
|
||||
info = yield self.get_json(uri, {})
|
||||
info = await self.get_json(uri, {})
|
||||
|
||||
if not _is_valid_3pe_metadata(info):
|
||||
logger.warning(
|
||||
@@ -202,14 +199,13 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
return None
|
||||
|
||||
key = (service.id, protocol)
|
||||
return self.protocol_meta_cache.wrap(key, _get)
|
||||
return await self.protocol_meta_cache.wrap(key, _get)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def push_bulk(self, service, events, txn_id=None):
|
||||
async def push_bulk(self, service, events, txn_id=None):
|
||||
if service.url is None:
|
||||
return True
|
||||
|
||||
events = self._serialize(events)
|
||||
events = self._serialize(service, events)
|
||||
|
||||
if txn_id is None:
|
||||
logger.warning(
|
||||
@@ -220,7 +216,7 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
|
||||
uri = service.url + ("/transactions/%s" % urllib.parse.quote(txn_id))
|
||||
try:
|
||||
yield self.put_json(
|
||||
await self.put_json(
|
||||
uri=uri,
|
||||
json_body={"events": events},
|
||||
args={"access_token": service.hs_token},
|
||||
@@ -235,6 +231,18 @@ class ApplicationServiceApi(SimpleHttpClient):
|
||||
failed_transactions_counter.labels(service.id).inc()
|
||||
return False
|
||||
|
||||
def _serialize(self, events):
|
||||
def _serialize(self, service, events):
|
||||
time_now = self.clock.time_msec()
|
||||
return [serialize_event(e, time_now, as_client_event=True) for e in events]
|
||||
return [
|
||||
serialize_event(
|
||||
e,
|
||||
time_now,
|
||||
as_client_event=True,
|
||||
is_invite=(
|
||||
e.type == EventTypes.Member
|
||||
and e.membership == "invite"
|
||||
and service.is_interested_in_user(e.state_key)
|
||||
),
|
||||
)
|
||||
for e in events
|
||||
]
|
||||
|
||||
@@ -50,8 +50,6 @@ components.
|
||||
"""
|
||||
import logging
|
||||
|
||||
from twisted.internet import defer
|
||||
|
||||
from synapse.appservice import ApplicationServiceState
|
||||
from synapse.logging.context import run_in_background
|
||||
from synapse.metrics.background_process_metrics import run_as_background_process
|
||||
@@ -59,7 +57,7 @@ from synapse.metrics.background_process_metrics import run_as_background_process
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ApplicationServiceScheduler(object):
|
||||
class ApplicationServiceScheduler:
|
||||
""" Public facing API for this module. Does the required DI to tie the
|
||||
components together. This also serves as the "event_pool", which in this
|
||||
case is a simple array.
|
||||
@@ -73,12 +71,11 @@ class ApplicationServiceScheduler(object):
|
||||
self.txn_ctrl = _TransactionController(self.clock, self.store, self.as_api)
|
||||
self.queuer = _ServiceQueuer(self.txn_ctrl, self.clock)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def start(self):
|
||||
async def start(self):
|
||||
logger.info("Starting appservice scheduler")
|
||||
|
||||
# check for any DOWN ASes and start recoverers for them.
|
||||
services = yield self.store.get_appservices_by_state(
|
||||
services = await self.store.get_appservices_by_state(
|
||||
ApplicationServiceState.DOWN
|
||||
)
|
||||
|
||||
@@ -89,7 +86,7 @@ class ApplicationServiceScheduler(object):
|
||||
self.queuer.enqueue(service, event)
|
||||
|
||||
|
||||
class _ServiceQueuer(object):
|
||||
class _ServiceQueuer:
|
||||
"""Queue of events waiting to be sent to appservices.
|
||||
|
||||
Groups events into transactions per-appservice, and sends them on to the
|
||||
@@ -117,8 +114,7 @@ class _ServiceQueuer(object):
|
||||
"as-sender-%s" % (service.id,), self._send_request, service
|
||||
)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _send_request(self, service):
|
||||
async def _send_request(self, service):
|
||||
# sanity-check: we shouldn't get here if this service already has a sender
|
||||
# running.
|
||||
assert service.id not in self.requests_in_flight
|
||||
@@ -130,14 +126,14 @@ class _ServiceQueuer(object):
|
||||
if not events:
|
||||
return
|
||||
try:
|
||||
yield self.txn_ctrl.send(service, events)
|
||||
await self.txn_ctrl.send(service, events)
|
||||
except Exception:
|
||||
logger.exception("AS request failed")
|
||||
finally:
|
||||
self.requests_in_flight.discard(service.id)
|
||||
|
||||
|
||||
class _TransactionController(object):
|
||||
class _TransactionController:
|
||||
"""Transaction manager.
|
||||
|
||||
Builds AppServiceTransactions and runs their lifecycle. Also starts a Recoverer
|
||||
@@ -162,36 +158,33 @@ class _TransactionController(object):
|
||||
# for UTs
|
||||
self.RECOVERER_CLASS = _Recoverer
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def send(self, service, events):
|
||||
async def send(self, service, events):
|
||||
try:
|
||||
txn = yield self.store.create_appservice_txn(service=service, events=events)
|
||||
service_is_up = yield self._is_service_up(service)
|
||||
txn = await self.store.create_appservice_txn(service=service, events=events)
|
||||
service_is_up = await self._is_service_up(service)
|
||||
if service_is_up:
|
||||
sent = yield txn.send(self.as_api)
|
||||
sent = await txn.send(self.as_api)
|
||||
if sent:
|
||||
yield txn.complete(self.store)
|
||||
await txn.complete(self.store)
|
||||
else:
|
||||
run_in_background(self._on_txn_fail, service)
|
||||
except Exception:
|
||||
logger.exception("Error creating appservice transaction")
|
||||
run_in_background(self._on_txn_fail, service)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def on_recovered(self, recoverer):
|
||||
async def on_recovered(self, recoverer):
|
||||
logger.info(
|
||||
"Successfully recovered application service AS ID %s", recoverer.service.id
|
||||
)
|
||||
self.recoverers.pop(recoverer.service.id)
|
||||
logger.info("Remaining active recoverers: %s", len(self.recoverers))
|
||||
yield self.store.set_appservice_state(
|
||||
await self.store.set_appservice_state(
|
||||
recoverer.service, ApplicationServiceState.UP
|
||||
)
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _on_txn_fail(self, service):
|
||||
async def _on_txn_fail(self, service):
|
||||
try:
|
||||
yield self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
|
||||
await self.store.set_appservice_state(service, ApplicationServiceState.DOWN)
|
||||
self.start_recoverer(service)
|
||||
except Exception:
|
||||
logger.exception("Error starting AS recoverer")
|
||||
@@ -211,13 +204,12 @@ class _TransactionController(object):
|
||||
recoverer.recover()
|
||||
logger.info("Now %i active recoverers", len(self.recoverers))
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def _is_service_up(self, service):
|
||||
state = yield self.store.get_appservice_state(service)
|
||||
async def _is_service_up(self, service):
|
||||
state = await self.store.get_appservice_state(service)
|
||||
return state == ApplicationServiceState.UP or state is None
|
||||
|
||||
|
||||
class _Recoverer(object):
|
||||
class _Recoverer:
|
||||
"""Manages retries and backoff for a DOWN appservice.
|
||||
|
||||
We have one of these for each appservice which is currently considered DOWN.
|
||||
@@ -254,25 +246,24 @@ class _Recoverer(object):
|
||||
self.backoff_counter += 1
|
||||
self.recover()
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def retry(self):
|
||||
async def retry(self):
|
||||
logger.info("Starting retries on %s", self.service.id)
|
||||
try:
|
||||
while True:
|
||||
txn = yield self.store.get_oldest_unsent_txn(self.service)
|
||||
txn = await self.store.get_oldest_unsent_txn(self.service)
|
||||
if not txn:
|
||||
# nothing left: we're done!
|
||||
self.callback(self)
|
||||
await self.callback(self)
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Retrying transaction %s for AS ID %s", txn.id, txn.service.id
|
||||
)
|
||||
sent = yield txn.send(self.as_api)
|
||||
sent = await txn.send(self.as_api)
|
||||
if not sent:
|
||||
break
|
||||
|
||||
yield txn.complete(self.store)
|
||||
await txn.complete(self.store)
|
||||
|
||||
# reset the backoff counter and then process the next transaction
|
||||
self.backoff_counter = 1
|
||||
|
||||
@@ -16,6 +16,7 @@ from synapse.config._base import ConfigError
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
from synapse.config.homeserver import HomeServerConfig
|
||||
|
||||
action = sys.argv[1]
|
||||
|
||||
@@ -18,12 +18,16 @@
|
||||
import argparse
|
||||
import errno
|
||||
import os
|
||||
import time
|
||||
import urllib.parse
|
||||
from collections import OrderedDict
|
||||
from hashlib import sha256
|
||||
from textwrap import dedent
|
||||
from typing import Any, MutableMapping, Optional
|
||||
|
||||
from six import integer_types
|
||||
from typing import Any, Callable, List, MutableMapping, Optional
|
||||
|
||||
import attr
|
||||
import jinja2
|
||||
import pkg_resources
|
||||
import yaml
|
||||
|
||||
|
||||
@@ -84,7 +88,7 @@ def path_exists(file_path):
|
||||
return False
|
||||
|
||||
|
||||
class Config(object):
|
||||
class Config:
|
||||
"""
|
||||
A configuration section, containing configuration keys and values.
|
||||
|
||||
@@ -100,6 +104,11 @@ class Config(object):
|
||||
def __init__(self, root_config=None):
|
||||
self.root = root_config
|
||||
|
||||
# Get the path to the default Synapse template directory
|
||||
self.default_template_dir = pkg_resources.resource_filename(
|
||||
"synapse", "res/templates"
|
||||
)
|
||||
|
||||
def __getattr__(self, item: str) -> Any:
|
||||
"""
|
||||
Try and fetch a configuration option that does not exist on this class.
|
||||
@@ -117,7 +126,7 @@ class Config(object):
|
||||
|
||||
@staticmethod
|
||||
def parse_size(value):
|
||||
if isinstance(value, integer_types):
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
sizes = {"K": 1024, "M": 1024 * 1024}
|
||||
size = 1
|
||||
@@ -129,7 +138,7 @@ class Config(object):
|
||||
|
||||
@staticmethod
|
||||
def parse_duration(value):
|
||||
if isinstance(value, integer_types):
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
second = 1000
|
||||
minute = 60 * second
|
||||
@@ -184,8 +193,97 @@ class Config(object):
|
||||
with open(file_path) as file_stream:
|
||||
return file_stream.read()
|
||||
|
||||
def read_templates(
|
||||
self, filenames: List[str], custom_template_directory: Optional[str] = None,
|
||||
) -> List[jinja2.Template]:
|
||||
"""Load a list of template files from disk using the given variables.
|
||||
|
||||
class RootConfig(object):
|
||||
This function will attempt to load the given templates from the default Synapse
|
||||
template directory. If `custom_template_directory` is supplied, that directory
|
||||
is tried first.
|
||||
|
||||
Files read are treated as Jinja templates. These templates are not rendered yet.
|
||||
|
||||
Args:
|
||||
filenames: A list of template filenames to read.
|
||||
|
||||
custom_template_directory: A directory to try to look for the templates
|
||||
before using the default Synapse template directory instead.
|
||||
|
||||
Raises:
|
||||
ConfigError: if the file's path is incorrect or otherwise cannot be read.
|
||||
|
||||
Returns:
|
||||
A list of jinja2 templates.
|
||||
"""
|
||||
templates = []
|
||||
search_directories = [self.default_template_dir]
|
||||
|
||||
# The loader will first look in the custom template directory (if specified) for the
|
||||
# given filename. If it doesn't find it, it will use the default template dir instead
|
||||
if custom_template_directory:
|
||||
# Check that the given template directory exists
|
||||
if not self.path_exists(custom_template_directory):
|
||||
raise ConfigError(
|
||||
"Configured template directory does not exist: %s"
|
||||
% (custom_template_directory,)
|
||||
)
|
||||
|
||||
# Search the custom template directory as well
|
||||
search_directories.insert(0, custom_template_directory)
|
||||
|
||||
loader = jinja2.FileSystemLoader(search_directories)
|
||||
env = jinja2.Environment(loader=loader, autoescape=True)
|
||||
|
||||
# Update the environment with our custom filters
|
||||
env.filters.update(
|
||||
{
|
||||
"format_ts": _format_ts_filter,
|
||||
"mxc_to_http": _create_mxc_to_http_filter(self.public_baseurl),
|
||||
}
|
||||
)
|
||||
|
||||
for filename in filenames:
|
||||
# Load the template
|
||||
template = env.get_template(filename)
|
||||
templates.append(template)
|
||||
|
||||
return templates
|
||||
|
||||
|
||||
def _format_ts_filter(value: int, format: str):
|
||||
return time.strftime(format, time.localtime(value / 1000))
|
||||
|
||||
|
||||
def _create_mxc_to_http_filter(public_baseurl: str) -> Callable:
|
||||
"""Create and return a jinja2 filter that converts MXC urls to HTTP
|
||||
|
||||
Args:
|
||||
public_baseurl: The public, accessible base URL of the homeserver
|
||||
"""
|
||||
|
||||
def mxc_to_http_filter(value, width, height, resize_method="crop"):
|
||||
if value[0:6] != "mxc://":
|
||||
return ""
|
||||
|
||||
server_and_media_id = value[6:]
|
||||
fragment = None
|
||||
if "#" in server_and_media_id:
|
||||
server_and_media_id, fragment = server_and_media_id.split("#", 1)
|
||||
fragment = "#" + fragment
|
||||
|
||||
params = {"width": width, "height": height, "method": resize_method}
|
||||
return "%s_matrix/media/v1/thumbnail/%s?%s%s" % (
|
||||
public_baseurl,
|
||||
server_and_media_id,
|
||||
urllib.parse.urlencode(params),
|
||||
fragment or "",
|
||||
)
|
||||
|
||||
return mxc_to_http_filter
|
||||
|
||||
|
||||
class RootConfig:
|
||||
"""
|
||||
Holder of an application's configuration.
|
||||
|
||||
@@ -719,4 +817,36 @@ def find_config_files(search_paths):
|
||||
return config_files
|
||||
|
||||
|
||||
__all__ = ["Config", "RootConfig"]
|
||||
@attr.s
|
||||
class ShardedWorkerHandlingConfig:
|
||||
"""Algorithm for choosing which instance is responsible for handling some
|
||||
sharded work.
|
||||
|
||||
For example, the federation senders use this to determine which instances
|
||||
handles sending stuff to a given destination (which is used as the `key`
|
||||
below).
|
||||
"""
|
||||
|
||||
instances = attr.ib(type=List[str])
|
||||
|
||||
def should_handle(self, instance_name: str, key: str) -> bool:
|
||||
"""Whether this instance is responsible for handling the given key.
|
||||
"""
|
||||
|
||||
# If multiple instances are not defined we always return true.
|
||||
if not self.instances or len(self.instances) == 1:
|
||||
return True
|
||||
|
||||
# We shard by taking the hash, modulo it by the number of instances and
|
||||
# then checking whether this instance matches the instance at that
|
||||
# index.
|
||||
#
|
||||
# (Technically this introduces some bias and is not entirely uniform,
|
||||
# but since the hash is so large the bias is ridiculously small).
|
||||
dest_hash = sha256(key.encode("utf8")).digest()
|
||||
dest_int = int.from_bytes(dest_hash, byteorder="little")
|
||||
remainder = dest_int % (len(self.instances))
|
||||
return self.instances[remainder] == instance_name
|
||||
|
||||
|
||||
__all__ = ["Config", "RootConfig", "ShardedWorkerHandlingConfig"]
|
||||
|
||||
@@ -137,3 +137,8 @@ class Config:
|
||||
|
||||
def read_config_files(config_files: List[str]): ...
|
||||
def find_config_files(search_paths: List[str]): ...
|
||||
|
||||
class ShardedWorkerHandlingConfig:
|
||||
instances: List[str]
|
||||
def __init__(self, instances: List[str]) -> None: ...
|
||||
def should_handle(self, instance_name: str, key: str) -> bool: ...
|
||||
|
||||
49
synapse/config/_util.py
Normal file
49
synapse/config/_util.py
Normal file
@@ -0,0 +1,49 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import Any, List
|
||||
|
||||
import jsonschema
|
||||
|
||||
from synapse.config._base import ConfigError
|
||||
from synapse.types import JsonDict
|
||||
|
||||
|
||||
def validate_config(json_schema: JsonDict, config: Any, config_path: List[str]) -> None:
|
||||
"""Validates a config setting against a JsonSchema definition
|
||||
|
||||
This can be used to validate a section of the config file against a schema
|
||||
definition. If the validation fails, a ConfigError is raised with a textual
|
||||
description of the problem.
|
||||
|
||||
Args:
|
||||
json_schema: the schema to validate against
|
||||
config: the configuration value to be validated
|
||||
config_path: the path within the config file. This will be used as a basis
|
||||
for the error message.
|
||||
"""
|
||||
try:
|
||||
jsonschema.validate(config, json_schema)
|
||||
except jsonschema.ValidationError as e:
|
||||
# copy `config_path` before modifying it.
|
||||
path = list(config_path)
|
||||
for p in list(e.path):
|
||||
if isinstance(p, int):
|
||||
path.append("<item %i>" % p)
|
||||
else:
|
||||
path.append(str(p))
|
||||
|
||||
raise ConfigError(
|
||||
"Unable to parse configuration: %s at %s" % (e.message, ".".join(path))
|
||||
)
|
||||
@@ -14,9 +14,7 @@
|
||||
|
||||
import logging
|
||||
from typing import Dict
|
||||
|
||||
from six import string_types
|
||||
from six.moves.urllib import parse as urlparse
|
||||
from urllib import parse as urlparse
|
||||
|
||||
import yaml
|
||||
from netaddr import IPSet
|
||||
@@ -98,17 +96,14 @@ def load_appservices(hostname, config_files):
|
||||
def _load_appservice(hostname, as_info, config_filename):
|
||||
required_string_fields = ["id", "as_token", "hs_token", "sender_localpart"]
|
||||
for field in required_string_fields:
|
||||
if not isinstance(as_info.get(field), string_types):
|
||||
if not isinstance(as_info.get(field), str):
|
||||
raise KeyError(
|
||||
"Required string field: '%s' (%s)" % (field, config_filename)
|
||||
)
|
||||
|
||||
# 'url' must either be a string or explicitly null, not missing
|
||||
# to avoid accidentally turning off push for ASes.
|
||||
if (
|
||||
not isinstance(as_info.get("url"), string_types)
|
||||
and as_info.get("url", "") is not None
|
||||
):
|
||||
if not isinstance(as_info.get("url"), str) and as_info.get("url", "") is not None:
|
||||
raise KeyError(
|
||||
"Required string field or explicit null: 'url' (%s)" % (config_filename,)
|
||||
)
|
||||
@@ -138,7 +133,7 @@ def _load_appservice(hostname, as_info, config_filename):
|
||||
ns,
|
||||
regex_obj,
|
||||
)
|
||||
if not isinstance(regex_obj.get("regex"), string_types):
|
||||
if not isinstance(regex_obj.get("regex"), str):
|
||||
raise ValueError("Missing/bad type 'regex' key in %s", regex_obj)
|
||||
if not isinstance(regex_obj.get("exclusive"), bool):
|
||||
raise ValueError(
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from typing import Callable, Dict
|
||||
|
||||
from ._base import Config, ConfigError
|
||||
@@ -25,11 +26,14 @@ _CACHE_PREFIX = "SYNAPSE_CACHE_FACTOR"
|
||||
# Map from canonicalised cache name to cache.
|
||||
_CACHES = {}
|
||||
|
||||
# a lock on the contents of _CACHES
|
||||
_CACHES_LOCK = threading.Lock()
|
||||
|
||||
_DEFAULT_FACTOR_SIZE = 0.5
|
||||
_DEFAULT_EVENT_CACHE_SIZE = "10K"
|
||||
|
||||
|
||||
class CacheProperties(object):
|
||||
class CacheProperties:
|
||||
def __init__(self):
|
||||
# The default factor size for all caches
|
||||
self.default_factor_size = float(
|
||||
@@ -66,7 +70,10 @@ def add_resizable_cache(cache_name: str, cache_resize_callback: Callable):
|
||||
# Some caches have '*' in them which we strip out.
|
||||
cache_name = _canonicalise_cache_name(cache_name)
|
||||
|
||||
_CACHES[cache_name] = cache_resize_callback
|
||||
# sometimes caches are initialised from background threads, so we need to make
|
||||
# sure we don't conflict with another thread running a resize operation
|
||||
with _CACHES_LOCK:
|
||||
_CACHES[cache_name] = cache_resize_callback
|
||||
|
||||
# Ensure all loaded caches are sized appropriately
|
||||
#
|
||||
@@ -87,7 +94,8 @@ class CacheConfig(Config):
|
||||
os.environ.get(_CACHE_PREFIX, _DEFAULT_FACTOR_SIZE)
|
||||
)
|
||||
properties.resize_all_caches_func = None
|
||||
_CACHES.clear()
|
||||
with _CACHES_LOCK:
|
||||
_CACHES.clear()
|
||||
|
||||
def generate_config_section(self, **kwargs):
|
||||
return """\
|
||||
@@ -193,6 +201,8 @@ class CacheConfig(Config):
|
||||
For each cache, run the mapped callback function with either
|
||||
a specific cache factor or the default, global one.
|
||||
"""
|
||||
for cache_name, callback in _CACHES.items():
|
||||
new_factor = self.cache_factors.get(cache_name, self.global_factor)
|
||||
callback(new_factor)
|
||||
# block other threads from modifying _CACHES while we iterate it.
|
||||
with _CACHES_LOCK:
|
||||
for cache_name, callback in _CACHES.items():
|
||||
new_factor = self.cache_factors.get(cache_name, self.global_factor)
|
||||
callback(new_factor)
|
||||
|
||||
@@ -55,7 +55,7 @@ DEFAULT_CONFIG = """\
|
||||
#database:
|
||||
# name: psycopg2
|
||||
# args:
|
||||
# user: synapse
|
||||
# user: synapse_user
|
||||
# password: secretpassword
|
||||
# database: synapse
|
||||
# host: localhost
|
||||
@@ -100,7 +100,10 @@ class DatabaseConnectionConfig:
|
||||
|
||||
self.name = name
|
||||
self.config = db_config
|
||||
self.data_stores = data_stores
|
||||
|
||||
# The `data_stores` config is actually talking about `databases` (we
|
||||
# changed the name).
|
||||
self.databases = data_stores
|
||||
|
||||
|
||||
class DatabaseConfig(Config):
|
||||
|
||||
@@ -14,7 +14,6 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
# This file can't be called email.py because if it is, we cannot:
|
||||
@@ -23,7 +22,7 @@ import os
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
|
||||
import pkg_resources
|
||||
import attr
|
||||
|
||||
from ._base import Config, ConfigError
|
||||
|
||||
@@ -33,6 +32,33 @@ Password reset emails are enabled on this homeserver due to a partial
|
||||
%s
|
||||
"""
|
||||
|
||||
DEFAULT_SUBJECTS = {
|
||||
"message_from_person_in_room": "[%(app)s] You have a message on %(app)s from %(person)s in the %(room)s room...",
|
||||
"message_from_person": "[%(app)s] You have a message on %(app)s from %(person)s...",
|
||||
"messages_from_person": "[%(app)s] You have messages on %(app)s from %(person)s...",
|
||||
"messages_in_room": "[%(app)s] You have messages on %(app)s in the %(room)s room...",
|
||||
"messages_in_room_and_others": "[%(app)s] You have messages on %(app)s in the %(room)s room and others...",
|
||||
"messages_from_person_and_others": "[%(app)s] You have messages on %(app)s from %(person)s and others...",
|
||||
"invite_from_person": "[%(app)s] %(person)s has invited you to chat on %(app)s...",
|
||||
"invite_from_person_to_room": "[%(app)s] %(person)s has invited you to join the %(room)s room on %(app)s...",
|
||||
"password_reset": "[%(server_name)s] Password reset",
|
||||
"email_validation": "[%(server_name)s] Validate your email",
|
||||
}
|
||||
|
||||
|
||||
@attr.s
|
||||
class EmailSubjectConfig:
|
||||
message_from_person_in_room = attr.ib(type=str)
|
||||
message_from_person = attr.ib(type=str)
|
||||
messages_from_person = attr.ib(type=str)
|
||||
messages_in_room = attr.ib(type=str)
|
||||
messages_in_room_and_others = attr.ib(type=str)
|
||||
messages_from_person_and_others = attr.ib(type=str)
|
||||
invite_from_person = attr.ib(type=str)
|
||||
invite_from_person_to_room = attr.ib(type=str)
|
||||
password_reset = attr.ib(type=str)
|
||||
email_validation = attr.ib(type=str)
|
||||
|
||||
|
||||
class EmailConfig(Config):
|
||||
section = "email"
|
||||
@@ -71,21 +97,18 @@ class EmailConfig(Config):
|
||||
if parsed[1] == "":
|
||||
raise RuntimeError("Invalid notif_from address")
|
||||
|
||||
# A user-configurable template directory
|
||||
template_dir = email_config.get("template_dir")
|
||||
# we need an absolute path, because we change directory after starting (and
|
||||
# we don't yet know what auxilliary templates like mail.css we will need).
|
||||
# (Note that loading as package_resources with jinja.PackageLoader doesn't
|
||||
# work for the same reason.)
|
||||
if not template_dir:
|
||||
template_dir = pkg_resources.resource_filename("synapse", "res/templates")
|
||||
|
||||
self.email_template_dir = os.path.abspath(template_dir)
|
||||
if isinstance(template_dir, str):
|
||||
# We need an absolute path, because we change directory after starting (and
|
||||
# we don't yet know what auxiliary templates like mail.css we will need).
|
||||
template_dir = os.path.abspath(template_dir)
|
||||
elif template_dir is not None:
|
||||
# If template_dir is something other than a str or None, warn the user
|
||||
raise ConfigError("Config option email.template_dir must be type str")
|
||||
|
||||
self.email_enable_notifs = email_config.get("enable_notifs", False)
|
||||
|
||||
account_validity_config = config.get("account_validity") or {}
|
||||
account_validity_renewal_enabled = account_validity_config.get("renew_at")
|
||||
|
||||
self.threepid_behaviour_email = (
|
||||
# Have Synapse handle the email sending if account_threepid_delegates.email
|
||||
# is not defined
|
||||
@@ -139,19 +162,6 @@ class EmailConfig(Config):
|
||||
email_config.get("validation_token_lifetime", "1h")
|
||||
)
|
||||
|
||||
if (
|
||||
self.email_enable_notifs
|
||||
or account_validity_renewal_enabled
|
||||
or self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
|
||||
):
|
||||
# make sure we can import the required deps
|
||||
import jinja2
|
||||
import bleach
|
||||
|
||||
# prevent unused warnings
|
||||
jinja2
|
||||
bleach
|
||||
|
||||
if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
|
||||
missing = []
|
||||
if not self.email_notif_from:
|
||||
@@ -169,49 +179,49 @@ class EmailConfig(Config):
|
||||
|
||||
# These email templates have placeholders in them, and thus must be
|
||||
# parsed using a templating engine during a request
|
||||
self.email_password_reset_template_html = email_config.get(
|
||||
password_reset_template_html = email_config.get(
|
||||
"password_reset_template_html", "password_reset.html"
|
||||
)
|
||||
self.email_password_reset_template_text = email_config.get(
|
||||
password_reset_template_text = email_config.get(
|
||||
"password_reset_template_text", "password_reset.txt"
|
||||
)
|
||||
self.email_registration_template_html = email_config.get(
|
||||
registration_template_html = email_config.get(
|
||||
"registration_template_html", "registration.html"
|
||||
)
|
||||
self.email_registration_template_text = email_config.get(
|
||||
registration_template_text = email_config.get(
|
||||
"registration_template_text", "registration.txt"
|
||||
)
|
||||
self.email_add_threepid_template_html = email_config.get(
|
||||
add_threepid_template_html = email_config.get(
|
||||
"add_threepid_template_html", "add_threepid.html"
|
||||
)
|
||||
self.email_add_threepid_template_text = email_config.get(
|
||||
add_threepid_template_text = email_config.get(
|
||||
"add_threepid_template_text", "add_threepid.txt"
|
||||
)
|
||||
|
||||
self.email_password_reset_template_failure_html = email_config.get(
|
||||
password_reset_template_failure_html = email_config.get(
|
||||
"password_reset_template_failure_html", "password_reset_failure.html"
|
||||
)
|
||||
self.email_registration_template_failure_html = email_config.get(
|
||||
registration_template_failure_html = email_config.get(
|
||||
"registration_template_failure_html", "registration_failure.html"
|
||||
)
|
||||
self.email_add_threepid_template_failure_html = email_config.get(
|
||||
add_threepid_template_failure_html = email_config.get(
|
||||
"add_threepid_template_failure_html", "add_threepid_failure.html"
|
||||
)
|
||||
|
||||
# These templates do not support any placeholder variables, so we
|
||||
# will read them from disk once during setup
|
||||
email_password_reset_template_success_html = email_config.get(
|
||||
password_reset_template_success_html = email_config.get(
|
||||
"password_reset_template_success_html", "password_reset_success.html"
|
||||
)
|
||||
email_registration_template_success_html = email_config.get(
|
||||
registration_template_success_html = email_config.get(
|
||||
"registration_template_success_html", "registration_success.html"
|
||||
)
|
||||
email_add_threepid_template_success_html = email_config.get(
|
||||
add_threepid_template_success_html = email_config.get(
|
||||
"add_threepid_template_success_html", "add_threepid_success.html"
|
||||
)
|
||||
|
||||
# Check templates exist
|
||||
for f in [
|
||||
# Read all templates from disk
|
||||
(
|
||||
self.email_password_reset_template_html,
|
||||
self.email_password_reset_template_text,
|
||||
self.email_registration_template_html,
|
||||
@@ -221,32 +231,36 @@ class EmailConfig(Config):
|
||||
self.email_password_reset_template_failure_html,
|
||||
self.email_registration_template_failure_html,
|
||||
self.email_add_threepid_template_failure_html,
|
||||
email_password_reset_template_success_html,
|
||||
email_registration_template_success_html,
|
||||
email_add_threepid_template_success_html,
|
||||
]:
|
||||
p = os.path.join(self.email_template_dir, f)
|
||||
if not os.path.isfile(p):
|
||||
raise ConfigError("Unable to find template file %s" % (p,))
|
||||
password_reset_template_success_html_template,
|
||||
registration_template_success_html_template,
|
||||
add_threepid_template_success_html_template,
|
||||
) = self.read_templates(
|
||||
[
|
||||
password_reset_template_html,
|
||||
password_reset_template_text,
|
||||
registration_template_html,
|
||||
registration_template_text,
|
||||
add_threepid_template_html,
|
||||
add_threepid_template_text,
|
||||
password_reset_template_failure_html,
|
||||
registration_template_failure_html,
|
||||
add_threepid_template_failure_html,
|
||||
password_reset_template_success_html,
|
||||
registration_template_success_html,
|
||||
add_threepid_template_success_html,
|
||||
],
|
||||
template_dir,
|
||||
)
|
||||
|
||||
# Retrieve content of web templates
|
||||
filepath = os.path.join(
|
||||
self.email_template_dir, email_password_reset_template_success_html
|
||||
# Render templates that do not contain any placeholders
|
||||
self.email_password_reset_template_success_html_content = (
|
||||
password_reset_template_success_html_template.render()
|
||||
)
|
||||
self.email_password_reset_template_success_html = self.read_file(
|
||||
filepath, "email.password_reset_template_success_html"
|
||||
self.email_registration_template_success_html_content = (
|
||||
registration_template_success_html_template.render()
|
||||
)
|
||||
filepath = os.path.join(
|
||||
self.email_template_dir, email_registration_template_success_html
|
||||
)
|
||||
self.email_registration_template_success_html_content = self.read_file(
|
||||
filepath, "email.registration_template_success_html"
|
||||
)
|
||||
filepath = os.path.join(
|
||||
self.email_template_dir, email_add_threepid_template_success_html
|
||||
)
|
||||
self.email_add_threepid_template_success_html_content = self.read_file(
|
||||
filepath, "email.add_threepid_template_success_html"
|
||||
self.email_add_threepid_template_success_html_content = (
|
||||
add_threepid_template_success_html_template.render()
|
||||
)
|
||||
|
||||
if self.email_enable_notifs:
|
||||
@@ -263,17 +277,19 @@ class EmailConfig(Config):
|
||||
% (", ".join(missing),)
|
||||
)
|
||||
|
||||
self.email_notif_template_html = email_config.get(
|
||||
notif_template_html = email_config.get(
|
||||
"notif_template_html", "notif_mail.html"
|
||||
)
|
||||
self.email_notif_template_text = email_config.get(
|
||||
notif_template_text = email_config.get(
|
||||
"notif_template_text", "notif_mail.txt"
|
||||
)
|
||||
|
||||
for f in self.email_notif_template_text, self.email_notif_template_html:
|
||||
p = os.path.join(self.email_template_dir, f)
|
||||
if not os.path.isfile(p):
|
||||
raise ConfigError("Unable to find email template file %s" % (p,))
|
||||
(
|
||||
self.email_notif_template_html,
|
||||
self.email_notif_template_text,
|
||||
) = self.read_templates(
|
||||
[notif_template_html, notif_template_text], template_dir,
|
||||
)
|
||||
|
||||
self.email_notif_for_new_users = email_config.get(
|
||||
"notif_for_new_users", True
|
||||
@@ -282,21 +298,32 @@ class EmailConfig(Config):
|
||||
"client_base_url", email_config.get("riot_base_url", None)
|
||||
)
|
||||
|
||||
if account_validity_renewal_enabled:
|
||||
self.email_expiry_template_html = email_config.get(
|
||||
if self.account_validity.renew_by_email_enabled:
|
||||
expiry_template_html = email_config.get(
|
||||
"expiry_template_html", "notice_expiry.html"
|
||||
)
|
||||
self.email_expiry_template_text = email_config.get(
|
||||
expiry_template_text = email_config.get(
|
||||
"expiry_template_text", "notice_expiry.txt"
|
||||
)
|
||||
|
||||
for f in self.email_expiry_template_text, self.email_expiry_template_html:
|
||||
p = os.path.join(self.email_template_dir, f)
|
||||
if not os.path.isfile(p):
|
||||
raise ConfigError("Unable to find email template file %s" % (p,))
|
||||
(
|
||||
self.account_validity_template_html,
|
||||
self.account_validity_template_text,
|
||||
) = self.read_templates(
|
||||
[expiry_template_html, expiry_template_text], template_dir,
|
||||
)
|
||||
|
||||
subjects_config = email_config.get("subjects", {})
|
||||
subjects = {}
|
||||
|
||||
for key, default in DEFAULT_SUBJECTS.items():
|
||||
subjects[key] = subjects_config.get(key, default)
|
||||
|
||||
self.email_subjects = EmailSubjectConfig(**subjects)
|
||||
|
||||
def generate_config_section(self, config_dir_path, server_name, **kwargs):
|
||||
return """\
|
||||
return (
|
||||
"""\
|
||||
# Configuration for sending emails from Synapse.
|
||||
#
|
||||
email:
|
||||
@@ -324,17 +351,17 @@ class EmailConfig(Config):
|
||||
# notif_from defines the "From" address to use when sending emails.
|
||||
# It must be set if email sending is enabled.
|
||||
#
|
||||
# The placeholder '%(app)s' will be replaced by the application name,
|
||||
# The placeholder '%%(app)s' will be replaced by the application name,
|
||||
# which is normally 'app_name' (below), but may be overridden by the
|
||||
# Matrix client application.
|
||||
#
|
||||
# Note that the placeholder must be written '%(app)s', including the
|
||||
# Note that the placeholder must be written '%%(app)s', including the
|
||||
# trailing 's'.
|
||||
#
|
||||
#notif_from: "Your Friendly %(app)s homeserver <noreply@example.com>"
|
||||
#notif_from: "Your Friendly %%(app)s homeserver <noreply@example.com>"
|
||||
|
||||
# app_name defines the default value for '%(app)s' in notif_from. It
|
||||
# defaults to 'Matrix'.
|
||||
# app_name defines the default value for '%%(app)s' in notif_from and email
|
||||
# subjects. It defaults to 'Matrix'.
|
||||
#
|
||||
#app_name: my_branded_matrix_server
|
||||
|
||||
@@ -364,9 +391,7 @@ class EmailConfig(Config):
|
||||
# Directory in which Synapse will try to find the template files below.
|
||||
# If not set, default templates from within the Synapse package will be used.
|
||||
#
|
||||
# DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
|
||||
# If you *do* uncomment it, you will need to make sure that all the templates
|
||||
# below are in the directory.
|
||||
# Do not uncomment this setting unless you want to customise the templates.
|
||||
#
|
||||
# Synapse will look for the following templates in this directory:
|
||||
#
|
||||
@@ -402,7 +427,76 @@ class EmailConfig(Config):
|
||||
# https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
|
||||
#
|
||||
#template_dir: "res/templates"
|
||||
|
||||
# Subjects to use when sending emails from Synapse.
|
||||
#
|
||||
# The placeholder '%%(app)s' will be replaced with the value of the 'app_name'
|
||||
# setting above, or by a value dictated by the Matrix client application.
|
||||
#
|
||||
# If a subject isn't overridden in this configuration file, the value used as
|
||||
# its example will be used.
|
||||
#
|
||||
#subjects:
|
||||
|
||||
# Subjects for notification emails.
|
||||
#
|
||||
# On top of the '%%(app)s' placeholder, these can use the following
|
||||
# placeholders:
|
||||
#
|
||||
# * '%%(person)s', which will be replaced by the display name of the user(s)
|
||||
# that sent the message(s), e.g. "Alice and Bob".
|
||||
# * '%%(room)s', which will be replaced by the name of the room the
|
||||
# message(s) have been sent to, e.g. "My super room".
|
||||
#
|
||||
# See the example provided for each setting to see which placeholder can be
|
||||
# used and how to use them.
|
||||
#
|
||||
# Subject to use to notify about one message from one or more user(s) in a
|
||||
# room which has a name.
|
||||
#message_from_person_in_room: "%(message_from_person_in_room)s"
|
||||
#
|
||||
# Subject to use to notify about one message from one or more user(s) in a
|
||||
# room which doesn't have a name.
|
||||
#message_from_person: "%(message_from_person)s"
|
||||
#
|
||||
# Subject to use to notify about multiple messages from one or more users in
|
||||
# a room which doesn't have a name.
|
||||
#messages_from_person: "%(messages_from_person)s"
|
||||
#
|
||||
# Subject to use to notify about multiple messages in a room which has a
|
||||
# name.
|
||||
#messages_in_room: "%(messages_in_room)s"
|
||||
#
|
||||
# Subject to use to notify about multiple messages in multiple rooms.
|
||||
#messages_in_room_and_others: "%(messages_in_room_and_others)s"
|
||||
#
|
||||
# Subject to use to notify about multiple messages from multiple persons in
|
||||
# multiple rooms. This is similar to the setting above except it's used when
|
||||
# the room in which the notification was triggered has no name.
|
||||
#messages_from_person_and_others: "%(messages_from_person_and_others)s"
|
||||
#
|
||||
# Subject to use to notify about an invite to a room which has a name.
|
||||
#invite_from_person_to_room: "%(invite_from_person_to_room)s"
|
||||
#
|
||||
# Subject to use to notify about an invite to a room which doesn't have a
|
||||
# name.
|
||||
#invite_from_person: "%(invite_from_person)s"
|
||||
|
||||
# Subject for emails related to account administration.
|
||||
#
|
||||
# On top of the '%%(app)s' placeholder, these one can use the
|
||||
# '%%(server_name)s' placeholder, which will be replaced by the value of the
|
||||
# 'server_name' setting in your Synapse configuration.
|
||||
#
|
||||
# Subject to use when sending a password reset email.
|
||||
#password_reset: "%(password_reset)s"
|
||||
#
|
||||
# Subject to use when sending a verification email to assert an address's
|
||||
# ownership.
|
||||
#email_validation: "%(email_validation)s"
|
||||
"""
|
||||
% DEFAULT_SUBJECTS
|
||||
)
|
||||
|
||||
|
||||
class ThreepidBehaviour(Enum):
|
||||
|
||||
88
synapse/config/federation.py
Normal file
88
synapse/config/federation.py
Normal file
@@ -0,0 +1,88 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from netaddr import IPSet
|
||||
|
||||
from ._base import Config, ConfigError
|
||||
|
||||
|
||||
class FederationConfig(Config):
|
||||
section = "federation"
|
||||
|
||||
def read_config(self, config, **kwargs):
|
||||
# FIXME: federation_domain_whitelist needs sytests
|
||||
self.federation_domain_whitelist = None # type: Optional[dict]
|
||||
federation_domain_whitelist = config.get("federation_domain_whitelist", None)
|
||||
|
||||
if federation_domain_whitelist is not None:
|
||||
# turn the whitelist into a hash for speed of lookup
|
||||
self.federation_domain_whitelist = {}
|
||||
|
||||
for domain in federation_domain_whitelist:
|
||||
self.federation_domain_whitelist[domain] = True
|
||||
|
||||
self.federation_ip_range_blacklist = config.get(
|
||||
"federation_ip_range_blacklist", []
|
||||
)
|
||||
|
||||
# Attempt to create an IPSet from the given ranges
|
||||
try:
|
||||
self.federation_ip_range_blacklist = IPSet(
|
||||
self.federation_ip_range_blacklist
|
||||
)
|
||||
|
||||
# Always blacklist 0.0.0.0, ::
|
||||
self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
|
||||
except Exception as e:
|
||||
raise ConfigError(
|
||||
"Invalid range(s) provided in federation_ip_range_blacklist: %s" % e
|
||||
)
|
||||
|
||||
def generate_config_section(self, config_dir_path, server_name, **kwargs):
|
||||
return """\
|
||||
# Restrict federation to the following whitelist of domains.
|
||||
# N.B. we recommend also firewalling your federation listener to limit
|
||||
# inbound federation traffic as early as possible, rather than relying
|
||||
# purely on this application-layer restriction. If not specified, the
|
||||
# default is to whitelist everything.
|
||||
#
|
||||
#federation_domain_whitelist:
|
||||
# - lon.example.com
|
||||
# - nyc.example.com
|
||||
# - syd.example.com
|
||||
|
||||
# Prevent federation requests from being sent to the following
|
||||
# blacklist IP address CIDR ranges. If this option is not specified, or
|
||||
# specified with an empty list, no ip range blacklist will be enforced.
|
||||
#
|
||||
# As of Synapse v1.4.0 this option also affects any outbound requests to identity
|
||||
# servers provided by user input.
|
||||
#
|
||||
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
|
||||
# listed here, since they correspond to unroutable addresses.)
|
||||
#
|
||||
federation_ip_range_blacklist:
|
||||
- '127.0.0.0/8'
|
||||
- '10.0.0.0/8'
|
||||
- '172.16.0.0/12'
|
||||
- '192.168.0.0/16'
|
||||
- '100.64.0.0/10'
|
||||
- '169.254.0.0/16'
|
||||
- '::1/128'
|
||||
- 'fe80::/64'
|
||||
- 'fc00::/7'
|
||||
"""
|
||||
@@ -23,6 +23,7 @@ from .cas import CasConfig
|
||||
from .consent_config import ConsentConfig
|
||||
from .database import DatabaseConfig
|
||||
from .emailconfig import EmailConfig
|
||||
from .federation import FederationConfig
|
||||
from .groups import GroupsConfig
|
||||
from .jwt_config import JWTConfig
|
||||
from .key import KeyConfig
|
||||
@@ -36,6 +37,7 @@ from .ratelimiting import RatelimitConfig
|
||||
from .redis import RedisConfig
|
||||
from .registration import RegistrationConfig
|
||||
from .repository import ContentRepositoryConfig
|
||||
from .room import RoomConfig
|
||||
from .room_directory import RoomDirectoryConfig
|
||||
from .saml2_config import SAML2Config
|
||||
from .server import ServerConfig
|
||||
@@ -56,6 +58,7 @@ class HomeServerConfig(RootConfig):
|
||||
config_classes = [
|
||||
ServerConfig,
|
||||
TlsConfig,
|
||||
FederationConfig,
|
||||
CacheConfig,
|
||||
DatabaseConfig,
|
||||
LoggingConfig,
|
||||
@@ -75,10 +78,10 @@ class HomeServerConfig(RootConfig):
|
||||
JWTConfig,
|
||||
PasswordConfig,
|
||||
EmailConfig,
|
||||
WorkerConfig,
|
||||
PasswordAuthProviderConfig,
|
||||
PushConfig,
|
||||
SpamCheckerConfig,
|
||||
RoomConfig,
|
||||
GroupsConfig,
|
||||
UserDirectoryConfig,
|
||||
ConsentConfig,
|
||||
@@ -87,5 +90,7 @@ class HomeServerConfig(RootConfig):
|
||||
RoomDirectoryConfig,
|
||||
ThirdPartyRulesConfig,
|
||||
TracerConfig,
|
||||
WorkerConfig,
|
||||
RedisConfig,
|
||||
FederationConfig,
|
||||
]
|
||||
|
||||
@@ -32,6 +32,11 @@ class JWTConfig(Config):
|
||||
self.jwt_secret = jwt_config["secret"]
|
||||
self.jwt_algorithm = jwt_config["algorithm"]
|
||||
|
||||
# The issuer and audiences are optional, if provided, it is asserted
|
||||
# that the claims exist on the JWT.
|
||||
self.jwt_issuer = jwt_config.get("issuer")
|
||||
self.jwt_audiences = jwt_config.get("audiences")
|
||||
|
||||
try:
|
||||
import jwt
|
||||
|
||||
@@ -42,13 +47,63 @@ class JWTConfig(Config):
|
||||
self.jwt_enabled = False
|
||||
self.jwt_secret = None
|
||||
self.jwt_algorithm = None
|
||||
self.jwt_issuer = None
|
||||
self.jwt_audiences = None
|
||||
|
||||
def generate_config_section(self, **kwargs):
|
||||
return """\
|
||||
# The JWT needs to contain a globally unique "sub" (subject) claim.
|
||||
# JSON web token integration. The following settings can be used to make
|
||||
# Synapse JSON web tokens for authentication, instead of its internal
|
||||
# password database.
|
||||
#
|
||||
# Each JSON Web Token needs to contain a "sub" (subject) claim, which is
|
||||
# used as the localpart of the mxid.
|
||||
#
|
||||
# Additionally, the expiration time ("exp"), not before time ("nbf"),
|
||||
# and issued at ("iat") claims are validated if present.
|
||||
#
|
||||
# Note that this is a non-standard login type and client support is
|
||||
# expected to be non-existant.
|
||||
#
|
||||
# See https://github.com/matrix-org/synapse/blob/master/docs/jwt.md.
|
||||
#
|
||||
#jwt_config:
|
||||
# enabled: true
|
||||
# secret: "a secret"
|
||||
# algorithm: "HS256"
|
||||
# Uncomment the following to enable authorization using JSON web
|
||||
# tokens. Defaults to false.
|
||||
#
|
||||
#enabled: true
|
||||
|
||||
# This is either the private shared secret or the public key used to
|
||||
# decode the contents of the JSON web token.
|
||||
#
|
||||
# Required if 'enabled' is true.
|
||||
#
|
||||
#secret: "provided-by-your-issuer"
|
||||
|
||||
# The algorithm used to sign the JSON web token.
|
||||
#
|
||||
# Supported algorithms are listed at
|
||||
# https://pyjwt.readthedocs.io/en/latest/algorithms.html
|
||||
#
|
||||
# Required if 'enabled' is true.
|
||||
#
|
||||
#algorithm: "provided-by-your-issuer"
|
||||
|
||||
# The issuer to validate the "iss" claim against.
|
||||
#
|
||||
# Optional, if provided the "iss" claim will be required and
|
||||
# validated for all JSON web tokens.
|
||||
#
|
||||
#issuer: "provided-by-your-issuer"
|
||||
|
||||
# A list of audiences to validate the "aud" claim against.
|
||||
#
|
||||
# Optional, if provided the "aud" claim will be required and
|
||||
# validated for all JSON web tokens.
|
||||
#
|
||||
# Note that if the "aud" claim is included in a JSON web token then
|
||||
# validation will fail without configuring audiences.
|
||||
#
|
||||
#audiences:
|
||||
# - "provided-by-your-issuer"
|
||||
"""
|
||||
|
||||
@@ -82,7 +82,7 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@attr.s
|
||||
class TrustedKeyServer(object):
|
||||
class TrustedKeyServer:
|
||||
# string: name of the server.
|
||||
server_name = attr.ib()
|
||||
|
||||
|
||||
@@ -55,24 +55,33 @@ formatters:
|
||||
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - \
|
||||
%(request)s - %(message)s'
|
||||
|
||||
filters:
|
||||
context:
|
||||
(): synapse.logging.context.LoggingContextFilter
|
||||
request: ""
|
||||
|
||||
handlers:
|
||||
file:
|
||||
class: logging.handlers.RotatingFileHandler
|
||||
class: logging.handlers.TimedRotatingFileHandler
|
||||
formatter: precise
|
||||
filename: ${log_file}
|
||||
maxBytes: 104857600
|
||||
backupCount: 10
|
||||
filters: [context]
|
||||
when: midnight
|
||||
backupCount: 3 # Does not include the current log file.
|
||||
encoding: utf8
|
||||
|
||||
# Default to buffering writes to log file for efficiency. This means that
|
||||
# will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
|
||||
# logs will still be flushed immediately.
|
||||
buffer:
|
||||
class: logging.handlers.MemoryHandler
|
||||
target: file
|
||||
# The capacity is the number of log lines that are buffered before
|
||||
# being written to disk. Increasing this will lead to better
|
||||
# performance, at the expensive of it taking longer for log lines to
|
||||
# be written to disk.
|
||||
capacity: 10
|
||||
flushLevel: 30 # Flush for WARNING logs as well
|
||||
|
||||
# A handler that writes logs to stderr. Unused by default, but can be used
|
||||
# instead of "buffer" and "file" in the logger handlers.
|
||||
console:
|
||||
class: logging.StreamHandler
|
||||
formatter: precise
|
||||
filters: [context]
|
||||
|
||||
loggers:
|
||||
synapse.storage.SQL:
|
||||
@@ -80,9 +89,24 @@ loggers:
|
||||
# information such as access tokens.
|
||||
level: INFO
|
||||
|
||||
twisted:
|
||||
# We send the twisted logging directly to the file handler,
|
||||
# to work around https://github.com/matrix-org/synapse/issues/3471
|
||||
# when using "buffer" logger. Use "console" to log to stderr instead.
|
||||
handlers: [file]
|
||||
propagate: false
|
||||
|
||||
root:
|
||||
level: INFO
|
||||
handlers: [file, console]
|
||||
|
||||
# Write logs to the `buffer` handler, which will buffer them together in memory,
|
||||
# then write them to a file.
|
||||
#
|
||||
# Replace "buffer" with "console" to log to stderr instead. (Note that you'll
|
||||
# also need to update the configuation for the `twisted` logger above, in
|
||||
# this case.)
|
||||
#
|
||||
handlers: [buffer]
|
||||
|
||||
disable_existing_loggers: false
|
||||
"""
|
||||
@@ -168,11 +192,26 @@ def _setup_stdlib_logging(config, log_config, logBeginner: LogBeginner):
|
||||
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(formatter)
|
||||
handler.addFilter(LoggingContextFilter(request=""))
|
||||
logger.addHandler(handler)
|
||||
else:
|
||||
logging.config.dictConfig(log_config)
|
||||
|
||||
# We add a log record factory that runs all messages through the
|
||||
# LoggingContextFilter so that we get the context *at the time we log*
|
||||
# rather than when we write to a handler. This can be done in config using
|
||||
# filter options, but care must when using e.g. MemoryHandler to buffer
|
||||
# writes.
|
||||
|
||||
log_filter = LoggingContextFilter(request="")
|
||||
old_factory = logging.getLogRecordFactory()
|
||||
|
||||
def factory(*args, **kwargs):
|
||||
record = old_factory(*args, **kwargs)
|
||||
log_filter.filter(record)
|
||||
return record
|
||||
|
||||
logging.setLogRecordFactory(factory)
|
||||
|
||||
# Route Twisted's native logging through to the standard library logging
|
||||
# system.
|
||||
observer = STDLibLogObserver()
|
||||
@@ -214,7 +253,7 @@ def setup_logging(
|
||||
Set up the logging subsystem.
|
||||
|
||||
Args:
|
||||
config (LoggingConfig | synapse.config.workers.WorkerConfig):
|
||||
config (LoggingConfig | synapse.config.worker.WorkerConfig):
|
||||
configuration data
|
||||
|
||||
use_worker_options (bool): True to use the 'worker_log_config' option
|
||||
|
||||
@@ -22,7 +22,7 @@ from ._base import Config, ConfigError
|
||||
|
||||
|
||||
@attr.s
|
||||
class MetricsFlags(object):
|
||||
class MetricsFlags:
|
||||
known_servers = attr.ib(default=False, validator=attr.validators.instance_of(bool))
|
||||
|
||||
@classmethod
|
||||
|
||||
@@ -89,7 +89,7 @@ class OIDCConfig(Config):
|
||||
# use an OpenID Connect Provider for authentication, instead of its internal
|
||||
# password database.
|
||||
#
|
||||
# See https://github.com/matrix-org/synapse/blob/master/openid.md.
|
||||
# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md.
|
||||
#
|
||||
oidc_config:
|
||||
# Uncomment the following to enable authorization against an OpenID Connect
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from ._base import Config
|
||||
from ._base import Config, ShardedWorkerHandlingConfig
|
||||
|
||||
|
||||
class PushConfig(Config):
|
||||
@@ -24,6 +24,9 @@ class PushConfig(Config):
|
||||
push_config = config.get("push", {})
|
||||
self.push_include_content = push_config.get("include_content", True)
|
||||
|
||||
pusher_instances = config.get("pusher_instances") or []
|
||||
self.pusher_shard_config = ShardedWorkerHandlingConfig(pusher_instances)
|
||||
|
||||
# There was a a 'redact_content' setting but mistakenly read from the
|
||||
# 'email'section'. Check for the flag in the 'push' section, and log,
|
||||
# but do not honour it to avoid nasty surprises when people upgrade.
|
||||
|
||||
@@ -17,7 +17,7 @@ from typing import Dict
|
||||
from ._base import Config
|
||||
|
||||
|
||||
class RateLimitConfig(object):
|
||||
class RateLimitConfig:
|
||||
def __init__(
|
||||
self,
|
||||
config: Dict[str, float],
|
||||
@@ -27,7 +27,7 @@ class RateLimitConfig(object):
|
||||
self.burst_count = config.get("burst_count", defaults["burst_count"])
|
||||
|
||||
|
||||
class FederationRateLimitConfig(object):
|
||||
class FederationRateLimitConfig:
|
||||
_items_and_default = {
|
||||
"window_size": 1000,
|
||||
"sleep_limit": 10,
|
||||
@@ -93,6 +93,15 @@ class RatelimitConfig(Config):
|
||||
if rc_admin_redaction:
|
||||
self.rc_admin_redaction = RateLimitConfig(rc_admin_redaction)
|
||||
|
||||
self.rc_joins_local = RateLimitConfig(
|
||||
config.get("rc_joins", {}).get("local", {}),
|
||||
defaults={"per_second": 0.1, "burst_count": 3},
|
||||
)
|
||||
self.rc_joins_remote = RateLimitConfig(
|
||||
config.get("rc_joins", {}).get("remote", {}),
|
||||
defaults={"per_second": 0.01, "burst_count": 3},
|
||||
)
|
||||
|
||||
def generate_config_section(self, **kwargs):
|
||||
return """\
|
||||
## Ratelimiting ##
|
||||
@@ -118,6 +127,10 @@ class RatelimitConfig(Config):
|
||||
# - one for ratelimiting redactions by room admins. If this is not explicitly
|
||||
# set then it uses the same ratelimiting as per rc_message. This is useful
|
||||
# to allow room admins to deal with abuse quickly.
|
||||
# - two for ratelimiting number of rooms a user can join, "local" for when
|
||||
# users are joining rooms the server is already in (this is cheap) vs
|
||||
# "remote" for when users are trying to join rooms not on the server (which
|
||||
# can be more expensive)
|
||||
#
|
||||
# The defaults are as shown below.
|
||||
#
|
||||
@@ -143,6 +156,14 @@ class RatelimitConfig(Config):
|
||||
#rc_admin_redaction:
|
||||
# per_second: 1
|
||||
# burst_count: 50
|
||||
#
|
||||
#rc_joins:
|
||||
# local:
|
||||
# per_second: 0.1
|
||||
# burst_count: 3
|
||||
# remote:
|
||||
# per_second: 0.01
|
||||
# burst_count: 3
|
||||
|
||||
|
||||
# Ratelimiting settings for incoming federation
|
||||
|
||||
@@ -21,7 +21,7 @@ class RedisConfig(Config):
|
||||
section = "redis"
|
||||
|
||||
def read_config(self, config, **kwargs):
|
||||
redis_config = config.get("redis", {})
|
||||
redis_config = config.get("redis") or {}
|
||||
self.redis_enabled = redis_config.get("enabled", False)
|
||||
|
||||
if not self.redis_enabled:
|
||||
@@ -32,3 +32,24 @@ class RedisConfig(Config):
|
||||
self.redis_host = redis_config.get("host", "localhost")
|
||||
self.redis_port = redis_config.get("port", 6379)
|
||||
self.redis_password = redis_config.get("password")
|
||||
|
||||
def generate_config_section(self, config_dir_path, server_name, **kwargs):
|
||||
return """\
|
||||
# Configuration for Redis when using workers. This *must* be enabled when
|
||||
# using workers (unless using old style direct TCP configuration).
|
||||
#
|
||||
redis:
|
||||
# Uncomment the below to enable Redis support.
|
||||
#
|
||||
#enabled: true
|
||||
|
||||
# Optional host and port to use to connect to redis. Defaults to
|
||||
# localhost and 6379
|
||||
#
|
||||
#host: localhost
|
||||
#port: 6379
|
||||
|
||||
# Optional password if configured on the Redis instance
|
||||
#
|
||||
#password: <secret_password>
|
||||
"""
|
||||
|
||||
@@ -18,8 +18,9 @@ from distutils.util import strtobool
|
||||
|
||||
import pkg_resources
|
||||
|
||||
from synapse.api.constants import RoomCreationPreset
|
||||
from synapse.config._base import Config, ConfigError
|
||||
from synapse.types import RoomAlias
|
||||
from synapse.types import RoomAlias, UserID
|
||||
from synapse.util.stringutils import random_string_with_symbols
|
||||
|
||||
|
||||
@@ -130,7 +131,50 @@ class RegistrationConfig(Config):
|
||||
for room_alias in self.auto_join_rooms:
|
||||
if not RoomAlias.is_valid(room_alias):
|
||||
raise ConfigError("Invalid auto_join_rooms entry %s" % (room_alias,))
|
||||
|
||||
# Options for creating auto-join rooms if they do not exist yet.
|
||||
self.autocreate_auto_join_rooms = config.get("autocreate_auto_join_rooms", True)
|
||||
self.autocreate_auto_join_rooms_federated = config.get(
|
||||
"autocreate_auto_join_rooms_federated", True
|
||||
)
|
||||
self.autocreate_auto_join_room_preset = (
|
||||
config.get("autocreate_auto_join_room_preset")
|
||||
or RoomCreationPreset.PUBLIC_CHAT
|
||||
)
|
||||
self.auto_join_room_requires_invite = self.autocreate_auto_join_room_preset in {
|
||||
RoomCreationPreset.PRIVATE_CHAT,
|
||||
RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
|
||||
}
|
||||
|
||||
# Pull the creater/inviter from the configuration, this gets used to
|
||||
# send invites for invite-only rooms.
|
||||
mxid_localpart = config.get("auto_join_mxid_localpart")
|
||||
self.auto_join_user_id = None
|
||||
if mxid_localpart:
|
||||
# Convert the localpart to a full mxid.
|
||||
self.auto_join_user_id = UserID(
|
||||
mxid_localpart, self.server_name
|
||||
).to_string()
|
||||
|
||||
if self.autocreate_auto_join_rooms:
|
||||
# Ensure the preset is a known value.
|
||||
if self.autocreate_auto_join_room_preset not in {
|
||||
RoomCreationPreset.PUBLIC_CHAT,
|
||||
RoomCreationPreset.PRIVATE_CHAT,
|
||||
RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
|
||||
}:
|
||||
raise ConfigError("Invalid value for autocreate_auto_join_room_preset")
|
||||
# If the preset requires invitations to be sent, ensure there's a
|
||||
# configured user to send them from.
|
||||
if self.auto_join_room_requires_invite:
|
||||
if not mxid_localpart:
|
||||
raise ConfigError(
|
||||
"The configuration option `auto_join_mxid_localpart` is required if "
|
||||
"`autocreate_auto_join_room_preset` is set to private_chat or trusted_private_chat, such that "
|
||||
"Synapse knows who to send invitations from. Please "
|
||||
"configure `auto_join_mxid_localpart`."
|
||||
)
|
||||
|
||||
self.auto_join_rooms_for_guests = config.get("auto_join_rooms_for_guests", True)
|
||||
|
||||
self.enable_set_displayname = config.get("enable_set_displayname", True)
|
||||
@@ -297,24 +341,6 @@ class RegistrationConfig(Config):
|
||||
#
|
||||
#default_identity_server: https://matrix.org
|
||||
|
||||
# The list of identity servers trusted to verify third party
|
||||
# identifiers by this server.
|
||||
#
|
||||
# Also defines the ID server which will be called when an account is
|
||||
# deactivated (one will be picked arbitrarily).
|
||||
#
|
||||
# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
|
||||
# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
|
||||
# background migration script, informing itself that the identity server all of its
|
||||
# 3PIDs have been bound to is likely one of the below.
|
||||
#
|
||||
# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
|
||||
# it is now solely used for the purposes of the background migration script, and can be
|
||||
# removed once it has run.
|
||||
#trusted_third_party_id_servers:
|
||||
# - matrix.org
|
||||
# - vector.im
|
||||
|
||||
# Handle threepid (email/phone etc) registration and password resets through a set of
|
||||
# *trusted* identity servers. Note that this allows the configured identity server to
|
||||
# reset passwords for accounts!
|
||||
@@ -365,7 +391,11 @@ class RegistrationConfig(Config):
|
||||
#enable_3pid_changes: false
|
||||
|
||||
# Users who register on this homeserver will automatically be joined
|
||||
# to these rooms
|
||||
# to these rooms.
|
||||
#
|
||||
# By default, any room aliases included in this list will be created
|
||||
# as a publicly joinable room when the first user registers for the
|
||||
# homeserver. This behaviour can be customised with the settings below.
|
||||
#
|
||||
#auto_join_rooms:
|
||||
# - "#example:example.com"
|
||||
@@ -373,10 +403,62 @@ class RegistrationConfig(Config):
|
||||
# Where auto_join_rooms are specified, setting this flag ensures that the
|
||||
# the rooms exist by creating them when the first user on the
|
||||
# homeserver registers.
|
||||
#
|
||||
# By default the auto-created rooms are publicly joinable from any federated
|
||||
# server. Use the autocreate_auto_join_rooms_federated and
|
||||
# autocreate_auto_join_room_preset settings below to customise this behaviour.
|
||||
#
|
||||
# Setting to false means that if the rooms are not manually created,
|
||||
# users cannot be auto-joined since they do not exist.
|
||||
#
|
||||
#autocreate_auto_join_rooms: true
|
||||
# Defaults to true. Uncomment the following line to disable automatically
|
||||
# creating auto-join rooms.
|
||||
#
|
||||
#autocreate_auto_join_rooms: false
|
||||
|
||||
# Whether the auto_join_rooms that are auto-created are available via
|
||||
# federation. Only has an effect if autocreate_auto_join_rooms is true.
|
||||
#
|
||||
# Note that whether a room is federated cannot be modified after
|
||||
# creation.
|
||||
#
|
||||
# Defaults to true: the room will be joinable from other servers.
|
||||
# Uncomment the following to prevent users from other homeservers from
|
||||
# joining these rooms.
|
||||
#
|
||||
#autocreate_auto_join_rooms_federated: false
|
||||
|
||||
# The room preset to use when auto-creating one of auto_join_rooms. Only has an
|
||||
# effect if autocreate_auto_join_rooms is true.
|
||||
#
|
||||
# This can be one of "public_chat", "private_chat", or "trusted_private_chat".
|
||||
# If a value of "private_chat" or "trusted_private_chat" is used then
|
||||
# auto_join_mxid_localpart must also be configured.
|
||||
#
|
||||
# Defaults to "public_chat", meaning that the room is joinable by anyone, including
|
||||
# federated servers if autocreate_auto_join_rooms_federated is true (the default).
|
||||
# Uncomment the following to require an invitation to join these rooms.
|
||||
#
|
||||
#autocreate_auto_join_room_preset: private_chat
|
||||
|
||||
# The local part of the user id which is used to create auto_join_rooms if
|
||||
# autocreate_auto_join_rooms is true. If this is not provided then the
|
||||
# initial user account that registers will be used to create the rooms.
|
||||
#
|
||||
# The user id is also used to invite new users to any auto-join rooms which
|
||||
# are set to invite-only.
|
||||
#
|
||||
# It *must* be configured if autocreate_auto_join_room_preset is set to
|
||||
# "private_chat" or "trusted_private_chat".
|
||||
#
|
||||
# Note that this must be specified in order for new users to be correctly
|
||||
# invited to any auto-join rooms which have been set to invite-only (either
|
||||
# at the time of creation or subsequently).
|
||||
#
|
||||
# Note that, if the room already exists, this user must be joined and
|
||||
# have the appropriate permissions to invite new members.
|
||||
#
|
||||
#auto_join_mxid_localpart: system
|
||||
|
||||
# When auto_join_rooms is specified, setting this flag to false prevents
|
||||
# guest accounts from being automatically joined to the rooms.
|
||||
|
||||
@@ -94,6 +94,12 @@ class ContentRepositoryConfig(Config):
|
||||
else:
|
||||
self.can_load_media_repo = True
|
||||
|
||||
# Whether this instance should be the one to run the background jobs to
|
||||
# e.g clean up old URL previews.
|
||||
self.media_instance_running_background_jobs = config.get(
|
||||
"media_instance_running_background_jobs",
|
||||
)
|
||||
|
||||
self.max_upload_size = self.parse_size(config.get("max_upload_size", "10M"))
|
||||
self.max_image_pixels = self.parse_size(config.get("max_image_pixels", "32M"))
|
||||
self.max_spider_size = self.parse_size(config.get("max_spider_size", "10M"))
|
||||
|
||||
85
synapse/config/room.py
Normal file
85
synapse/config/room.py
Normal file
@@ -0,0 +1,85 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 The Matrix.org Foundation C.I.C.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
|
||||
from synapse.api.constants import RoomCreationPreset
|
||||
|
||||
from ._base import Config, ConfigError
|
||||
|
||||
logger = logging.Logger(__name__)
|
||||
|
||||
|
||||
class RoomDefaultEncryptionTypes:
|
||||
"""Possible values for the encryption_enabled_by_default_for_room_type config option"""
|
||||
|
||||
ALL = "all"
|
||||
INVITE = "invite"
|
||||
OFF = "off"
|
||||
|
||||
|
||||
class RoomConfig(Config):
|
||||
section = "room"
|
||||
|
||||
def read_config(self, config, **kwargs):
|
||||
# Whether new, locally-created rooms should have encryption enabled
|
||||
encryption_for_room_type = config.get(
|
||||
"encryption_enabled_by_default_for_room_type",
|
||||
RoomDefaultEncryptionTypes.OFF,
|
||||
)
|
||||
if encryption_for_room_type == RoomDefaultEncryptionTypes.ALL:
|
||||
self.encryption_enabled_by_default_for_room_presets = [
|
||||
RoomCreationPreset.PRIVATE_CHAT,
|
||||
RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
|
||||
RoomCreationPreset.PUBLIC_CHAT,
|
||||
]
|
||||
elif encryption_for_room_type == RoomDefaultEncryptionTypes.INVITE:
|
||||
self.encryption_enabled_by_default_for_room_presets = [
|
||||
RoomCreationPreset.PRIVATE_CHAT,
|
||||
RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
|
||||
]
|
||||
elif (
|
||||
encryption_for_room_type == RoomDefaultEncryptionTypes.OFF
|
||||
or encryption_for_room_type is False
|
||||
):
|
||||
# PyYAML translates "off" into False if it's unquoted, so we also need to
|
||||
# check for encryption_for_room_type being False.
|
||||
self.encryption_enabled_by_default_for_room_presets = []
|
||||
else:
|
||||
raise ConfigError(
|
||||
"Invalid value for encryption_enabled_by_default_for_room_type"
|
||||
)
|
||||
|
||||
def generate_config_section(self, **kwargs):
|
||||
return """\
|
||||
## Rooms ##
|
||||
|
||||
# Controls whether locally-created rooms should be end-to-end encrypted by
|
||||
# default.
|
||||
#
|
||||
# Possible options are "all", "invite", and "off". They are defined as:
|
||||
#
|
||||
# * "all": any locally-created room
|
||||
# * "invite": any room created with the "private_chat" or "trusted_private_chat"
|
||||
# room creation presets
|
||||
# * "off": this option will take no effect
|
||||
#
|
||||
# The default value is "off".
|
||||
#
|
||||
# Note that this option will only affect rooms created after it is set. It
|
||||
# will also not affect rooms created by other servers.
|
||||
#
|
||||
#encryption_enabled_by_default_for_room_type: invite
|
||||
"""
|
||||
@@ -149,7 +149,7 @@ class RoomDirectoryConfig(Config):
|
||||
return False
|
||||
|
||||
|
||||
class _RoomDirectoryRule(object):
|
||||
class _RoomDirectoryRule:
|
||||
"""Helper class to test whether a room directory action is allowed, like
|
||||
creating an alias or publishing a room.
|
||||
"""
|
||||
|
||||
@@ -15,14 +15,15 @@
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
from typing import Any, List
|
||||
|
||||
import jinja2
|
||||
import pkg_resources
|
||||
import attr
|
||||
|
||||
from synapse.python_dependencies import DependencyException, check_requirements
|
||||
from synapse.util.module_loader import load_module, load_python_module
|
||||
|
||||
from ._base import Config, ConfigError
|
||||
from ._util import validate_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -80,6 +81,11 @@ class SAML2Config(Config):
|
||||
|
||||
self.saml2_enabled = True
|
||||
|
||||
attribute_requirements = saml2_config.get("attribute_requirements") or []
|
||||
self.attribute_requirements = _parse_attribute_requirements_def(
|
||||
attribute_requirements
|
||||
)
|
||||
|
||||
self.saml2_grandfathered_mxid_source_attribute = saml2_config.get(
|
||||
"grandfathered_mxid_source_attribute", "uid"
|
||||
)
|
||||
@@ -160,18 +166,12 @@ class SAML2Config(Config):
|
||||
|
||||
# session lifetime: in milliseconds
|
||||
self.saml2_session_lifetime = self.parse_duration(
|
||||
saml2_config.get("saml_session_lifetime", "5m")
|
||||
saml2_config.get("saml_session_lifetime", "15m")
|
||||
)
|
||||
|
||||
template_dir = saml2_config.get("template_dir")
|
||||
if not template_dir:
|
||||
template_dir = pkg_resources.resource_filename("synapse", "res/templates",)
|
||||
|
||||
loader = jinja2.FileSystemLoader(template_dir)
|
||||
# enable auto-escape here, to having to remember to escape manually in the
|
||||
# template
|
||||
env = jinja2.Environment(loader=loader, autoescape=True)
|
||||
self.saml2_error_html_template = env.get_template("saml_error.html")
|
||||
self.saml2_error_html_template = self.read_templates(
|
||||
["saml_error.html"], saml2_config.get("template_dir")
|
||||
)[0]
|
||||
|
||||
def _default_saml_config_dict(
|
||||
self, required_attributes: set, optional_attributes: set
|
||||
@@ -286,7 +286,7 @@ class SAML2Config(Config):
|
||||
|
||||
# The lifetime of a SAML session. This defines how long a user has to
|
||||
# complete the authentication process, if allow_unsolicited is unset.
|
||||
# The default is 5 minutes.
|
||||
# The default is 15 minutes.
|
||||
#
|
||||
#saml_session_lifetime: 5m
|
||||
|
||||
@@ -341,6 +341,17 @@ class SAML2Config(Config):
|
||||
#
|
||||
#grandfathered_mxid_source_attribute: upn
|
||||
|
||||
# It is possible to configure Synapse to only allow logins if SAML attributes
|
||||
# match particular values. The requirements can be listed under
|
||||
# `attribute_requirements` as shown below. All of the listed attributes must
|
||||
# match for the login to be permitted.
|
||||
#
|
||||
#attribute_requirements:
|
||||
# - attribute: userGroup
|
||||
# value: "staff"
|
||||
# - attribute: department
|
||||
# value: "sales"
|
||||
|
||||
# Directory in which Synapse will try to find the template files below.
|
||||
# If not set, default templates from within the Synapse package will be used.
|
||||
#
|
||||
@@ -368,3 +379,34 @@ class SAML2Config(Config):
|
||||
""" % {
|
||||
"config_dir_path": config_dir_path
|
||||
}
|
||||
|
||||
|
||||
@attr.s(frozen=True)
|
||||
class SamlAttributeRequirement:
|
||||
"""Object describing a single requirement for SAML attributes."""
|
||||
|
||||
attribute = attr.ib(type=str)
|
||||
value = attr.ib(type=str)
|
||||
|
||||
JSON_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {"attribute": {"type": "string"}, "value": {"type": "string"}},
|
||||
"required": ["attribute", "value"],
|
||||
}
|
||||
|
||||
|
||||
ATTRIBUTE_REQUIREMENTS_SCHEMA = {
|
||||
"type": "array",
|
||||
"items": SamlAttributeRequirement.JSON_SCHEMA,
|
||||
}
|
||||
|
||||
|
||||
def _parse_attribute_requirements_def(
|
||||
attribute_requirements: Any,
|
||||
) -> List[SamlAttributeRequirement]:
|
||||
validate_config(
|
||||
ATTRIBUTE_REQUIREMENTS_SCHEMA,
|
||||
attribute_requirements,
|
||||
config_path=["saml2_config", "attribute_requirements"],
|
||||
)
|
||||
return [SamlAttributeRequirement(**x) for x in attribute_requirements]
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user