Merge branch 'develop' into dmr/pyproject-poetry
Signed-off-by: Dan Callahan <danc@element.io>
This commit is contained in:
Dan Callahan
2
.github/workflows/docs.yaml
vendored
2
.github/workflows/docs.yaml
vendored
@@ -22,7 +22,7 @@ jobs:
|
||||
- name: Setup mdbook
|
||||
uses: peaceiris/actions-mdbook@4b5ef36b314c2599664ca107bb8c02412548d79d # v1.1.14
|
||||
with:
|
||||
mdbook-version: '0.4.9'
|
||||
mdbook-version: '0.4.17'
|
||||
|
||||
- name: Build the documentation
|
||||
# mdbook will only create an index.html if we're including docs/README.md in SUMMARY.md.
|
||||
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
@@ -15,7 +15,8 @@ _trial_temp*/
|
||||
.DS_Store
|
||||
__pycache__/
|
||||
|
||||
# We do want the poetry lockfile. Ignoring '*.lock' above seems to ignore tox and trial lock files.
|
||||
# We do want the poetry lockfile.
|
||||
# TODO: is there a good reason for ignoring '*.lock' above? If not, nuke it.
|
||||
!poetry.lock
|
||||
|
||||
# stuff that is likely to exist when you run a server locally
|
||||
|
||||
3640
CHANGES-pre-1.0.md
Normal file
3640
CHANGES-pre-1.0.md
Normal file
File diff suppressed because it is too large
Load Diff
3725
CHANGES.md
3725
CHANGES.md
File diff suppressed because it is too large
Load Diff
@@ -1 +0,0 @@
|
||||
Rename `shared_rooms` to `mutual_rooms` (MSC2666), as per proposal changes.
|
||||
@@ -1 +0,0 @@
|
||||
Remove check on `update_user_directory` for shared rooms handler (MSC2666), and update/expand documentation.
|
||||
1
changelog.d/12040.feature
Normal file
1
changelog.d/12040.feature
Normal file
@@ -0,0 +1 @@
|
||||
Optimise fetching large quantities of missing room state over federation.
|
||||
@@ -1 +0,0 @@
|
||||
Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) instead of abusing `auth_event_ids`.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a long-standing bug which caused the `/_matrix/federation/v1/state` and `.../state_ids` endpoints to return incorrect or invalid data when called for an event which we have stored as an "outlier".
|
||||
@@ -1 +0,0 @@
|
||||
Refuse to start if registration is enabled without email, captcha, or token-based verification unless new config flag `enable_registration_without_verification` is set.
|
||||
@@ -1 +0,0 @@
|
||||
Allow modules to store already existing 3PID associations.
|
||||
@@ -1 +0,0 @@
|
||||
Add tests for database transaction callbacks.
|
||||
@@ -1 +0,0 @@
|
||||
Handle cancellation in `DatabasePool.runInteraction()`.
|
||||
1
changelog.d/12209.misc
Normal file
1
changelog.d/12209.misc
Normal file
@@ -0,0 +1 @@
|
||||
Switch to using a sequence to generate AS transaction IDs. Contributed by Nick Beeper. If running synapse with a dedicated appservice worker, this MUST be stopped before upgrading the main process and database.
|
||||
@@ -1 +0,0 @@
|
||||
Add missing type hints for cache storage.
|
||||
@@ -1 +0,0 @@
|
||||
Clean-up logic around rebasing URLs for URL image previews.
|
||||
@@ -1 +0,0 @@
|
||||
Add type hints to tests files.
|
||||
@@ -1 +0,0 @@
|
||||
Use the `ignored_users` table in additional places instead of re-parsing the account data.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a long-standing bug where events from ignored users were still considered for relations.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a bug introduced in v1.53.0 where an unnecessary query could be performed when fetching bundled aggregations for threads.
|
||||
@@ -1 +0,0 @@
|
||||
Fix the link to the module documentation in the legacy spam checker warning message.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a long-standing bug where events from ignored users were still considered for relations.
|
||||
@@ -1 +0,0 @@
|
||||
Refactor the relations endpoints to add a `RelationsHandler`.
|
||||
@@ -1 +0,0 @@
|
||||
Generate announcement links in the release script.
|
||||
@@ -1 +0,0 @@
|
||||
Remove incorrect prefixes in the worker documentation for some endpoints.
|
||||
@@ -1 +0,0 @@
|
||||
Improve error message when dependencies check finds a broken installation.
|
||||
@@ -1 +0,0 @@
|
||||
Correct `check_username_for_spam` annotations and docs.
|
||||
@@ -1 +0,0 @@
|
||||
Add missing type hints for storage.
|
||||
@@ -1 +0,0 @@
|
||||
Allow registering admin users using the module API. Contributed by Famedly.
|
||||
1
changelog.d/12251.feature
Normal file
1
changelog.d/12251.feature
Normal file
@@ -0,0 +1 @@
|
||||
Offload the `update_client_ip` background job from the main process to the background worker, when using Redis-based replication.
|
||||
@@ -1 +0,0 @@
|
||||
Add missing type hints for storage.
|
||||
@@ -1 +0,0 @@
|
||||
Add type hints to tests files.
|
||||
@@ -1 +0,0 @@
|
||||
Compress metrics HTTP resource when enabled. Contributed by Nick @ Beeper.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a bug introduced in Synapse 1.52 where admins could not deactivate and GDPR-erase a user if Synapse was configured with limits on avatars.
|
||||
@@ -1 +0,0 @@
|
||||
Refuse to start if DB has non-`C` locale, unless config flag `allow_unsafe_db_locale` is set to true.
|
||||
@@ -1 +0,0 @@
|
||||
Optionally include account validity expiration information to experimental [MSC3720](https://github.com/matrix-org/matrix-doc/pull/3720) account status responses.
|
||||
1
changelog.d/12267.misc
Normal file
1
changelog.d/12267.misc
Normal file
@@ -0,0 +1 @@
|
||||
Add missing type hints for storage.
|
||||
@@ -1 +0,0 @@
|
||||
Use type stubs for `psycopg2`.
|
||||
@@ -1 +0,0 @@
|
||||
Add a new cache `_get_membership_from_event_id` to speed up push rule calculations in large rooms.
|
||||
@@ -1 +0,0 @@
|
||||
Corrected Authentik OpenID typo, added helpful note for troubleshooting. Contributed by @IronTooch.
|
||||
@@ -1 +0,0 @@
|
||||
HAProxy reverse proxy guide update to stop sending IPv4-mapped address to homeserver. Contributed by @villepeh.
|
||||
@@ -1 +0,0 @@
|
||||
Re-enable Complement concurrency in CI.
|
||||
@@ -1 +0,0 @@
|
||||
Fix a long-standing bug where events from ignored users were still considered for relations.
|
||||
@@ -1 +0,0 @@
|
||||
Refuse to start if DB has non-`C` locale, unless config flag `allow_unsafe_db_locale` is set to true.
|
||||
@@ -1 +0,0 @@
|
||||
Remove unused test utilities.
|
||||
1
changelog.d/12295.misc
Normal file
1
changelog.d/12295.misc
Normal file
@@ -0,0 +1 @@
|
||||
Move [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654) support behind an experimental configuration flag.
|
||||
@@ -1 +0,0 @@
|
||||
Enhance logging for inbound federation events.
|
||||
@@ -1 +0,0 @@
|
||||
Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) instead of abusing `auth_event_ids`.
|
||||
@@ -1 +0,0 @@
|
||||
Improve type annotations for `execute_values`.
|
||||
@@ -1 +0,0 @@
|
||||
Fix compatibility with the recently-released Jinja 3.1.
|
||||
1
changelog.d/12317.misc
Normal file
1
changelog.d/12317.misc
Normal file
@@ -0,0 +1 @@
|
||||
Update docstrings to explain how to decipher live and historic pagination tokens.
|
||||
@@ -1 +0,0 @@
|
||||
Bump the version of `black` for compatibility with the latest `click` release.
|
||||
1
changelog.d/12321.misc
Normal file
1
changelog.d/12321.misc
Normal file
@@ -0,0 +1 @@
|
||||
Add ground work for speeding up device list updates for users in large numbers of rooms.
|
||||
1
changelog.d/12327.feature
Normal file
1
changelog.d/12327.feature
Normal file
@@ -0,0 +1 @@
|
||||
Add a module callback to react to account data changes.
|
||||
1
changelog.d/12329.bugfix
Normal file
1
changelog.d/12329.bugfix
Normal file
@@ -0,0 +1 @@
|
||||
Fix non-member state events not resolving for historical events when used in [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716) `/batch_send` `state_events_at_start`.
|
||||
1
changelog.d/12332.misc
Normal file
1
changelog.d/12332.misc
Normal file
@@ -0,0 +1 @@
|
||||
Avoid trying to calculate the state at outlier events.
|
||||
1
changelog.d/12339.doc
Normal file
1
changelog.d/12339.doc
Normal file
@@ -0,0 +1 @@
|
||||
Upgrade the version of `mdbook` in CI to 0.4.17.
|
||||
1
changelog.d/12341.feature
Normal file
1
changelog.d/12341.feature
Normal file
@@ -0,0 +1 @@
|
||||
Allow setting user admin status using the module API. Contributed by Famedly.
|
||||
1
changelog.d/12345.doc
Normal file
1
changelog.d/12345.doc
Normal file
@@ -0,0 +1 @@
|
||||
Updates to the Room DAG concepts development document to clarify that we mark events as outliers because we don't have any state for them.
|
||||
1
changelog.d/12346.misc
Normal file
1
changelog.d/12346.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove redundant `get_success` calls in test code.
|
||||
1
changelog.d/12347.misc
Normal file
1
changelog.d/12347.misc
Normal file
@@ -0,0 +1 @@
|
||||
Add type annotations for `tests/unittest.py`.
|
||||
1
changelog.d/12348.misc
Normal file
1
changelog.d/12348.misc
Normal file
@@ -0,0 +1 @@
|
||||
Move single-use methods out of `TestCase`.
|
||||
1
changelog.d/12349.misc
Normal file
1
changelog.d/12349.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove broken and unused development scripts.
|
||||
1
changelog.d/12350.bugfix
Normal file
1
changelog.d/12350.bugfix
Normal file
@@ -0,0 +1 @@
|
||||
Default to `private` room visibility rather than `public` when a client does not specify one, according to spec.
|
||||
1
changelog.d/12351.misc
Normal file
1
changelog.d/12351.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove broken and unused development scripts.
|
||||
1
changelog.d/12353.misc
Normal file
1
changelog.d/12353.misc
Normal file
@@ -0,0 +1 @@
|
||||
Convert `Linearizer` tests from `inlineCallbacks` to async.
|
||||
1
changelog.d/12354.misc
Normal file
1
changelog.d/12354.misc
Normal file
@@ -0,0 +1 @@
|
||||
Update docstrings for `ReadWriteLock` tests.
|
||||
1
changelog.d/12355.misc
Normal file
1
changelog.d/12355.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove broken and unused development scripts.
|
||||
1
changelog.d/12357.misc
Normal file
1
changelog.d/12357.misc
Normal file
@@ -0,0 +1 @@
|
||||
Refactor `Linearizer`, convert methods to async and use an async context manager.
|
||||
1
changelog.d/12358.misc
Normal file
1
changelog.d/12358.misc
Normal file
@@ -0,0 +1 @@
|
||||
Fix a long-standing bug where `Linearizer`s could get stuck if a cancellation were to happen at the wrong time.
|
||||
1
changelog.d/12364.bugfix
Normal file
1
changelog.d/12364.bugfix
Normal file
@@ -0,0 +1 @@
|
||||
Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string.
|
||||
1
changelog.d/12366.misc
Normal file
1
changelog.d/12366.misc
Normal file
@@ -0,0 +1 @@
|
||||
Make `StreamToken.from_string` and `RoomStreamToken.parse` propagate cancellations instead of replacing them with `SynapseError`s.
|
||||
1
changelog.d/12367.feature
Normal file
1
changelog.d/12367.feature
Normal file
@@ -0,0 +1 @@
|
||||
Reduce overhead of restarting synchrotrons.
|
||||
1
changelog.d/12369.doc
Normal file
1
changelog.d/12369.doc
Normal file
@@ -0,0 +1 @@
|
||||
Update the link to Redis pub/sub documentation in the workers documentation..
|
||||
1
changelog.d/12370.feature
Normal file
1
changelog.d/12370.feature
Normal file
@@ -0,0 +1 @@
|
||||
Update `/messages` to use historic pagination tokens if no `from` query parameter is given.
|
||||
1
changelog.d/12372.feature
Normal file
1
changelog.d/12372.feature
Normal file
@@ -0,0 +1 @@
|
||||
Reduce overhead of restarting synchrotrons.
|
||||
1
changelog.d/12376.misc
Normal file
1
changelog.d/12376.misc
Normal file
@@ -0,0 +1 @@
|
||||
Allow specifying the Postgres database's port when running unit tests with Postgres.
|
||||
1
changelog.d/12379.misc
Normal file
1
changelog.d/12379.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove temporary pin of signedjson<=1.1.1 that was added in Synapse 1.56.0.
|
||||
1
changelog.d/12381.misc
Normal file
1
changelog.d/12381.misc
Normal file
@@ -0,0 +1 @@
|
||||
Lay groundwork for using `poetry` to manage Synapse's dependencies.
|
||||
1
changelog.d/12384.misc
Normal file
1
changelog.d/12384.misc
Normal file
@@ -0,0 +1 @@
|
||||
Make missing `importlib_metadata` dependency explicit.
|
||||
1
changelog.d/12389.misc
Normal file
1
changelog.d/12389.misc
Normal file
@@ -0,0 +1 @@
|
||||
Update type annotations for compatiblity with prometheus_client 0.14.
|
||||
1
changelog.d/12390.bugfix
Normal file
1
changelog.d/12390.bugfix
Normal file
@@ -0,0 +1 @@
|
||||
Fix a bug introduced in Synapse 1.49.0 which caused the `synapse_event_persisted_position` metric to have invalid values.
|
||||
1
changelog.d/12392.doc
Normal file
1
changelog.d/12392.doc
Normal file
@@ -0,0 +1 @@
|
||||
Remove documentation for converting a legacy structured logging configuration to the new format.
|
||||
1
changelog.d/12398.misc
Normal file
1
changelog.d/12398.misc
Normal file
@@ -0,0 +1 @@
|
||||
Remove support for the unstable identifiers specified in [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).
|
||||
1
changelog.d/12400.misc
Normal file
1
changelog.d/12400.misc
Normal file
@@ -0,0 +1 @@
|
||||
Make missing `importlib_metadata` dependency explicit.
|
||||
14
debian/changelog
vendored
14
debian/changelog
vendored
@@ -1,10 +1,22 @@
|
||||
matrix-synapse-py3 (1.55.2+nmu1) UNRELEASED; urgency=medium
|
||||
matrix-synapse-py3 (1.56.0+nmu1) UNRELEASED; urgency=medium
|
||||
|
||||
* Non-maintainer upload.
|
||||
* Use poetry to manage the bundled virtualenv included with this package.
|
||||
|
||||
-- Synapse Packaging Team <packages@matrix.org> Wed, 30 Mar 2022 12:21:43 +0100
|
||||
|
||||
matrix-synapse-py3 (1.56.0) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.56.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 05 Apr 2022 12:38:39 +0100
|
||||
|
||||
matrix-synapse-py3 (1.56.0~rc1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.56.0~rc1.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 29 Mar 2022 10:40:50 +0100
|
||||
|
||||
matrix-synapse-py3 (1.55.2) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.55.2.
|
||||
|
||||
@@ -45,6 +45,7 @@
|
||||
- [Account validity callbacks](modules/account_validity_callbacks.md)
|
||||
- [Password auth provider callbacks](modules/password_auth_provider_callbacks.md)
|
||||
- [Background update controller callbacks](modules/background_update_controller_callbacks.md)
|
||||
- [Account data callbacks](modules/account_data_callbacks.md)
|
||||
- [Porting a legacy module to the new interface](modules/porting_legacy_module.md)
|
||||
- [Workers](workers.md)
|
||||
- [Using `synctl` with Workers](synctl_workers.md)
|
||||
|
||||
@@ -201,9 +201,10 @@ To do so, [configure Postgres](../postgres.md) and run `trial` with the
|
||||
following environment variables matching your configuration:
|
||||
|
||||
- `SYNAPSE_POSTGRES` to anything nonempty
|
||||
- `SYNAPSE_POSTGRES_HOST`
|
||||
- `SYNAPSE_POSTGRES_USER`
|
||||
- `SYNAPSE_POSTGRES_PASSWORD`
|
||||
- `SYNAPSE_POSTGRES_HOST` (optional if it's the default: UNIX socket)
|
||||
- `SYNAPSE_POSTGRES_PORT` (optional if it's the default: 5432)
|
||||
- `SYNAPSE_POSTGRES_USER` (optional if using a UNIX socket)
|
||||
- `SYNAPSE_POSTGRES_PASSWORD` (optional if using a UNIX socket)
|
||||
|
||||
For example:
|
||||
|
||||
@@ -215,6 +216,13 @@ export SYNAPSE_POSTGRES_PASSWORD=mydevenvpassword
|
||||
trial
|
||||
```
|
||||
|
||||
You don't need to specify the host, user, port or password if your Postgres
|
||||
server is set to authenticate you over the UNIX socket (i.e. if the `psql` command
|
||||
works without further arguments).
|
||||
|
||||
Your Postgres account needs to be able to create databases.
|
||||
|
||||
|
||||
## Run the integration tests ([Sytest](https://github.com/matrix-org/sytest)).
|
||||
|
||||
The integration tests are a more comprehensive suite of tests. They
|
||||
|
||||
@@ -39,7 +39,8 @@ yet correlated to the DAG.
|
||||
Outliers typically arise when we fetch the auth chain or state for a given
|
||||
event. When that happens, we just grab the events in the state/auth chain,
|
||||
without calculating the state at those events, or backfilling their
|
||||
`prev_events`.
|
||||
`prev_events`. Since we don't have the state at any events fetched in that
|
||||
way, we mark them as outliers.
|
||||
|
||||
So, typically, we won't have the `prev_events` of an `outlier` in the database,
|
||||
(though it's entirely possible that we *might* have them for some other
|
||||
|
||||
106
docs/modules/account_data_callbacks.md
Normal file
106
docs/modules/account_data_callbacks.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Account data callbacks
|
||||
|
||||
Account data callbacks allow module developers to react to changes of the account data
|
||||
of local users. Account data callbacks can be registered using the module API's
|
||||
`register_account_data_callbacks` method.
|
||||
|
||||
## Callbacks
|
||||
|
||||
The available account data callbacks are:
|
||||
|
||||
### `on_account_data_updated`
|
||||
|
||||
_First introduced in Synapse v1.57.0_
|
||||
|
||||
```python
|
||||
async def on_account_data_updated(
|
||||
user_id: str,
|
||||
room_id: Optional[str],
|
||||
account_data_type: str,
|
||||
content: "synapse.module_api.JsonDict",
|
||||
) -> None:
|
||||
```
|
||||
|
||||
Called after user's account data has been updated. The module is given the
|
||||
Matrix ID of the user whose account data is changing, the room ID the data is associated
|
||||
with, the type associated with the change, as well as the new content. If the account
|
||||
data is not associated with a specific room, then the room ID is `None`.
|
||||
|
||||
This callback is triggered when new account data is added or when the data associated with
|
||||
a given type (and optionally room) changes. This includes deletion, since in Matrix,
|
||||
deleting account data consists of replacing the data associated with a given type
|
||||
(and optionally room) with an empty dictionary (`{}`).
|
||||
|
||||
Note that this doesn't trigger when changing the tags associated with a room, as these are
|
||||
processed separately by Synapse.
|
||||
|
||||
If multiple modules implement this callback, Synapse runs them all in order.
|
||||
|
||||
## Example
|
||||
|
||||
The example below is a module that implements the `on_account_data_updated` callback, and
|
||||
sends an event to an audit room when a user changes their account data.
|
||||
|
||||
```python
|
||||
import json
|
||||
import attr
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from synapse.module_api import JsonDict, ModuleApi
|
||||
from synapse.module_api.errors import ConfigError
|
||||
|
||||
|
||||
@attr.s(auto_attribs=True)
|
||||
class CustomAccountDataConfig:
|
||||
audit_room: str
|
||||
sender: str
|
||||
|
||||
|
||||
class CustomAccountDataModule:
|
||||
def __init__(self, config: CustomAccountDataConfig, api: ModuleApi):
|
||||
self.api = api
|
||||
self.config = config
|
||||
|
||||
self.api.register_account_data_callbacks(
|
||||
on_account_data_updated=self.log_new_account_data,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def parse_config(config: Dict[str, Any]) -> CustomAccountDataConfig:
|
||||
def check_in_config(param: str):
|
||||
if param not in config:
|
||||
raise ConfigError(f"'{param}' is required")
|
||||
|
||||
check_in_config("audit_room")
|
||||
check_in_config("sender")
|
||||
|
||||
return CustomAccountDataConfig(
|
||||
audit_room=config["audit_room"],
|
||||
sender=config["sender"],
|
||||
)
|
||||
|
||||
async def log_new_account_data(
|
||||
self,
|
||||
user_id: str,
|
||||
room_id: Optional[str],
|
||||
account_data_type: str,
|
||||
content: JsonDict,
|
||||
) -> None:
|
||||
content_raw = json.dumps(content)
|
||||
msg_content = f"{user_id} has changed their account data for type {account_data_type} to: {content_raw}"
|
||||
|
||||
if room_id is not None:
|
||||
msg_content += f" (in room {room_id})"
|
||||
|
||||
await self.api.create_and_send_event_into_room(
|
||||
{
|
||||
"room_id": self.config.audit_room,
|
||||
"sender": self.config.sender,
|
||||
"type": "m.room.message",
|
||||
"content": {
|
||||
"msgtype": "m.text",
|
||||
"body": msg_content
|
||||
}
|
||||
}
|
||||
)
|
||||
```
|
||||
@@ -33,7 +33,7 @@ A module can implement the following static method:
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def parse_config(config: dict) -> dict
|
||||
def parse_config(config: dict) -> Any
|
||||
```
|
||||
|
||||
This method is given a dictionary resulting from parsing the YAML configuration for the
|
||||
|
||||
@@ -78,82 +78,3 @@ loggers:
|
||||
The above logging config will set Synapse as 'INFO' logging level by default,
|
||||
with the SQL layer at 'WARNING', and will log JSON formatted messages to a
|
||||
remote endpoint at 10.1.2.3:9999.
|
||||
|
||||
## Upgrading from legacy structured logging configuration
|
||||
|
||||
Versions of Synapse prior to v1.54.0 automatically converted the legacy
|
||||
structured logging configuration, which was deprecated in v1.23.0, to the standard
|
||||
library logging configuration.
|
||||
|
||||
The following reference can be used to update your configuration. Based on the
|
||||
drain `type`, we can pick a new handler:
|
||||
|
||||
1. For a type of `console`, `console_json`, or `console_json_terse`: a handler
|
||||
with a class of `logging.StreamHandler` and a `stream` of `ext://sys.stdout`
|
||||
or `ext://sys.stderr` should be used.
|
||||
2. For a type of `file` or `file_json`: a handler of `logging.FileHandler` with
|
||||
a location of the file path should be used.
|
||||
3. For a type of `network_json_terse`: a handler of `synapse.logging.RemoteHandler`
|
||||
with the host and port should be used.
|
||||
|
||||
Then based on the drain `type` we can pick a new formatter:
|
||||
|
||||
1. For a type of `console` or `file` no formatter is necessary.
|
||||
2. For a type of `console_json` or `file_json`: a formatter of
|
||||
`synapse.logging.JsonFormatter` should be used.
|
||||
3. For a type of `console_json_terse` or `network_json_terse`: a formatter of
|
||||
`synapse.logging.TerseJsonFormatter` should be used.
|
||||
|
||||
For each new handler and formatter they should be added to the logging configuration
|
||||
and then assigned to either a logger or the root logger.
|
||||
|
||||
An example legacy configuration:
|
||||
|
||||
```yaml
|
||||
structured: true
|
||||
|
||||
loggers:
|
||||
synapse:
|
||||
level: INFO
|
||||
synapse.storage.SQL:
|
||||
level: WARNING
|
||||
|
||||
drains:
|
||||
console:
|
||||
type: console
|
||||
location: stdout
|
||||
file:
|
||||
type: file_json
|
||||
location: homeserver.log
|
||||
```
|
||||
|
||||
Would be converted into a new configuration:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
|
||||
formatters:
|
||||
json:
|
||||
class: synapse.logging.JsonFormatter
|
||||
|
||||
handlers:
|
||||
console:
|
||||
class: logging.StreamHandler
|
||||
stream: ext://sys.stdout
|
||||
file:
|
||||
class: logging.FileHandler
|
||||
formatter: json
|
||||
filename: homeserver.log
|
||||
|
||||
loggers:
|
||||
synapse:
|
||||
level: INFO
|
||||
handlers: [console, file]
|
||||
synapse.storage.SQL:
|
||||
level: WARNING
|
||||
```
|
||||
|
||||
The new logging configuration is a bit more verbose, but significantly more
|
||||
flexible. It allows for configuration that were not previously possible, such as
|
||||
sending plain logs over the network, or using different handlers for different
|
||||
modules.
|
||||
|
||||
@@ -88,8 +88,26 @@ process, for example:
|
||||
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
```
|
||||
|
||||
# Upgrading to v1.57.0
|
||||
|
||||
## Changes to database schema for application services
|
||||
|
||||
Synapse v1.57.0 includes a [change](https://github.com/matrix-org/synapse/pull/12209) to the
|
||||
way transaction IDs are managed for application services. If your deployment uses a dedicated
|
||||
worker for application service traffic, **it must be stopped** when the database is upgraded
|
||||
(which normally happens when the main process is upgraded), to ensure the change is made safely
|
||||
without any risk of reusing transaction IDs.
|
||||
|
||||
Deployments which do not use separate worker processes can be upgraded as normal. Similarly,
|
||||
deployments where no applciation services are in use can be upgraded as normal.
|
||||
|
||||
# Upgrading to v1.56.0
|
||||
|
||||
## Open registration without verification is now disabled by default
|
||||
|
||||
Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config
|
||||
flag `enable_registration_without_verification` is set to "true".
|
||||
|
||||
## Groups/communities feature has been deprecated
|
||||
|
||||
The non-standard groups/communities feature in Synapse has been deprecated and will
|
||||
@@ -111,12 +129,6 @@ for more information and instructions on how to fix a database with incorrect va
|
||||
|
||||
# Upgrading to v1.55.0
|
||||
|
||||
## Open registration without verification is now disabled by default
|
||||
|
||||
Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config
|
||||
flag `enable_registration_without_verification` is set to "true".
|
||||
|
||||
|
||||
## `synctl` script has been moved
|
||||
|
||||
The `synctl` script
|
||||
@@ -151,7 +163,7 @@ please upgrade Mjolnir to version 1.3.2 or later before upgrading Synapse.
|
||||
This release removes support for the `structured: true` logging configuration
|
||||
which was deprecated in Synapse v1.23.0. If your logging configuration contains
|
||||
`structured: true` then it should be modified based on the
|
||||
[structured logging documentation](structured_logging.md).
|
||||
[structured logging documentation](https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration).
|
||||
|
||||
# Upgrading to v1.53.0
|
||||
|
||||
@@ -768,7 +780,7 @@ lock down external access to the Admin API endpoints.
|
||||
This release deprecates use of the `structured: true` logging
|
||||
configuration for structured logging. If your logging configuration
|
||||
contains `structured: true` then it should be modified based on the
|
||||
[structured logging documentation](structured_logging.md).
|
||||
[structured logging documentation](https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration).
|
||||
|
||||
The `structured` and `drains` logging options are now deprecated and
|
||||
should be replaced by standard logging configuration of `handlers` and
|
||||
|
||||
@@ -27,7 +27,7 @@ feeds streams of newly written data between processes so they can be kept in
|
||||
sync with the database state.
|
||||
|
||||
When configured to do so, Synapse uses a
|
||||
[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication
|
||||
[Redis pub/sub channel](https://redis.io/docs/manual/pubsub/) to send the replication
|
||||
stream between all configured Synapse processes. Additionally, processes may
|
||||
make HTTP requests to each other, primarily for operations which need to wait
|
||||
for a reply ─ such as sending an event.
|
||||
|
||||
8
mypy.ini
8
mypy.ini
@@ -23,13 +23,8 @@ files =
|
||||
exclude = (?x)
|
||||
^(
|
||||
|scripts-dev/build_debian_packages.py
|
||||
|scripts-dev/check_signature.py
|
||||
|scripts-dev/definitions.py
|
||||
|scripts-dev/federation_client.py
|
||||
|scripts-dev/hash_history.py
|
||||
|scripts-dev/list_url_patterns.py
|
||||
|scripts-dev/release.py
|
||||
|scripts-dev/tail-synapse.py
|
||||
|
||||
|synapse/_scripts/export_signing_key.py
|
||||
|synapse/_scripts/move_remote_media_to_new_store.py
|
||||
@@ -42,7 +37,6 @@ exclude = (?x)
|
||||
|synapse/storage/databases/main/event_federation.py
|
||||
|synapse/storage/databases/main/push_rule.py
|
||||
|synapse/storage/databases/main/roommember.py
|
||||
|synapse/storage/databases/main/state.py
|
||||
|synapse/storage/schema/
|
||||
|
||||
|tests/api/test_auth.py
|
||||
@@ -81,11 +75,9 @@ exclude = (?x)
|
||||
|tests/storage/test_base.py
|
||||
|tests/storage/test_roommember.py
|
||||
|tests/test_metrics.py
|
||||
|tests/test_phone_home.py
|
||||
|tests/test_server.py
|
||||
|tests/test_state.py
|
||||
|tests/test_terms_auth.py
|
||||
|tests/unittest.py
|
||||
|tests/util/caches/test_cached_call.py
|
||||
|tests/util/caches/test_deferred_cache.py
|
||||
|tests/util/caches/test_descriptors.py
|
||||
|
||||
@@ -54,7 +54,7 @@ skip_gitignore = true
|
||||
|
||||
[tool.poetry]
|
||||
name = "matrix-synapse"
|
||||
version = "1.54.0rc1"
|
||||
version = "1.56.0"
|
||||
description = "Homeserver for the Matrix decentralised comms protocol"
|
||||
authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
|
||||
license = "Apache-2.0"
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import dns.resolver
|
||||
import urllib2
|
||||
from signedjson.key import decode_verify_key_bytes, write_signing_keys
|
||||
from signedjson.sign import verify_signed_json
|
||||
from unpaddedbase64 import decode_base64
|
||||
|
||||
|
||||
def get_targets(server_name):
|
||||
if ":" in server_name:
|
||||
target, port = server_name.split(":")
|
||||
yield (target, int(port))
|
||||
return
|
||||
try:
|
||||
answers = dns.resolver.query("_matrix._tcp." + server_name, "SRV")
|
||||
for srv in answers:
|
||||
yield (srv.target, srv.port)
|
||||
except dns.resolver.NXDOMAIN:
|
||||
yield (server_name, 8448)
|
||||
|
||||
|
||||
def get_server_keys(server_name, target, port):
|
||||
url = "https://%s:%i/_matrix/key/v1" % (target, port)
|
||||
keys = json.load(urllib2.urlopen(url))
|
||||
verify_keys = {}
|
||||
for key_id, key_base64 in keys["verify_keys"].items():
|
||||
verify_key = decode_verify_key_bytes(key_id, decode_base64(key_base64))
|
||||
verify_signed_json(keys, server_name, verify_key)
|
||||
verify_keys[key_id] = verify_key
|
||||
return verify_keys
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("signature_name")
|
||||
parser.add_argument(
|
||||
"input_json", nargs="?", type=argparse.FileType("r"), default=sys.stdin
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig()
|
||||
|
||||
server_name = args.signature_name
|
||||
keys = {}
|
||||
for target, port in get_targets(server_name):
|
||||
try:
|
||||
keys = get_server_keys(server_name, target, port)
|
||||
print("Using keys from https://%s:%s/_matrix/key/v1" % (target, port))
|
||||
write_signing_keys(sys.stdout, keys.values())
|
||||
break
|
||||
except Exception:
|
||||
logging.exception("Error talking to %s:%s", target, port)
|
||||
|
||||
json_to_check = json.load(args.input_json)
|
||||
print("Checking JSON:")
|
||||
for key_id in json_to_check["signatures"][args.signature_name]:
|
||||
try:
|
||||
key = keys[key_id]
|
||||
verify_signed_json(json_to_check, args.signature_name, key)
|
||||
print("PASS %s" % (key_id,))
|
||||
except Exception:
|
||||
logging.exception("Check for key %s failed" % (key_id,))
|
||||
print("FAIL %s" % (key_id,))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,208 +0,0 @@
|
||||
#! /usr/bin/python
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
class DefinitionVisitor(ast.NodeVisitor):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.functions = {}
|
||||
self.classes = {}
|
||||
self.names = {}
|
||||
self.attrs = set()
|
||||
self.definitions = {
|
||||
"def": self.functions,
|
||||
"class": self.classes,
|
||||
"names": self.names,
|
||||
"attrs": self.attrs,
|
||||
}
|
||||
|
||||
def visit_Name(self, node):
|
||||
self.names.setdefault(type(node.ctx).__name__, set()).add(node.id)
|
||||
|
||||
def visit_Attribute(self, node):
|
||||
self.attrs.add(node.attr)
|
||||
for child in ast.iter_child_nodes(node):
|
||||
self.visit(child)
|
||||
|
||||
def visit_ClassDef(self, node):
|
||||
visitor = DefinitionVisitor()
|
||||
self.classes[node.name] = visitor.definitions
|
||||
for child in ast.iter_child_nodes(node):
|
||||
visitor.visit(child)
|
||||
|
||||
def visit_FunctionDef(self, node):
|
||||
visitor = DefinitionVisitor()
|
||||
self.functions[node.name] = visitor.definitions
|
||||
for child in ast.iter_child_nodes(node):
|
||||
visitor.visit(child)
|
||||
|
||||
|
||||
def non_empty(defs):
|
||||
functions = {name: non_empty(f) for name, f in defs["def"].items()}
|
||||
classes = {name: non_empty(f) for name, f in defs["class"].items()}
|
||||
result = {}
|
||||
if functions:
|
||||
result["def"] = functions
|
||||
if classes:
|
||||
result["class"] = classes
|
||||
names = defs["names"]
|
||||
uses = []
|
||||
for name in names.get("Load", ()):
|
||||
if name not in names.get("Param", ()) and name not in names.get("Store", ()):
|
||||
uses.append(name)
|
||||
uses.extend(defs["attrs"])
|
||||
if uses:
|
||||
result["uses"] = uses
|
||||
result["names"] = names
|
||||
result["attrs"] = defs["attrs"]
|
||||
return result
|
||||
|
||||
|
||||
def definitions_in_code(input_code):
|
||||
input_ast = ast.parse(input_code)
|
||||
visitor = DefinitionVisitor()
|
||||
visitor.visit(input_ast)
|
||||
definitions = non_empty(visitor.definitions)
|
||||
return definitions
|
||||
|
||||
|
||||
def definitions_in_file(filepath):
|
||||
with open(filepath) as f:
|
||||
return definitions_in_code(f.read())
|
||||
|
||||
|
||||
def defined_names(prefix, defs, names):
|
||||
for name, funcs in defs.get("def", {}).items():
|
||||
names.setdefault(name, {"defined": []})["defined"].append(prefix + name)
|
||||
defined_names(prefix + name + ".", funcs, names)
|
||||
|
||||
for name, funcs in defs.get("class", {}).items():
|
||||
names.setdefault(name, {"defined": []})["defined"].append(prefix + name)
|
||||
defined_names(prefix + name + ".", funcs, names)
|
||||
|
||||
|
||||
def used_names(prefix, item, defs, names):
|
||||
for name, funcs in defs.get("def", {}).items():
|
||||
used_names(prefix + name + ".", name, funcs, names)
|
||||
|
||||
for name, funcs in defs.get("class", {}).items():
|
||||
used_names(prefix + name + ".", name, funcs, names)
|
||||
|
||||
path = prefix.rstrip(".")
|
||||
for used in defs.get("uses", ()):
|
||||
if used in names:
|
||||
if item:
|
||||
names[item].setdefault("uses", []).append(used)
|
||||
names[used].setdefault("used", {}).setdefault(item, []).append(path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
parser = argparse.ArgumentParser(description="Find definitions.")
|
||||
parser.add_argument(
|
||||
"--unused", action="store_true", help="Only list unused definitions"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ignore", action="append", metavar="REGEXP", help="Ignore a pattern"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pattern", action="append", metavar="REGEXP", help="Search for a pattern"
|
||||
)
|
||||
parser.add_argument(
|
||||
"directories",
|
||||
nargs="+",
|
||||
metavar="DIR",
|
||||
help="Directories to search for definitions",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--referrers",
|
||||
default=0,
|
||||
type=int,
|
||||
help="Include referrers up to the given depth",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--referred",
|
||||
default=0,
|
||||
type=int,
|
||||
help="Include referred down to the given depth",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--format", default="yaml", help="Output format, one of 'yaml' or 'dot'"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
definitions = {}
|
||||
for directory in args.directories:
|
||||
for root, _, files in os.walk(directory):
|
||||
for filename in files:
|
||||
if filename.endswith(".py"):
|
||||
filepath = os.path.join(root, filename)
|
||||
definitions[filepath] = definitions_in_file(filepath)
|
||||
|
||||
names = {}
|
||||
for filepath, defs in definitions.items():
|
||||
defined_names(filepath + ":", defs, names)
|
||||
|
||||
for filepath, defs in definitions.items():
|
||||
used_names(filepath + ":", None, defs, names)
|
||||
|
||||
patterns = [re.compile(pattern) for pattern in args.pattern or ()]
|
||||
ignore = [re.compile(pattern) for pattern in args.ignore or ()]
|
||||
|
||||
result = {}
|
||||
for name, definition in names.items():
|
||||
if patterns and not any(pattern.match(name) for pattern in patterns):
|
||||
continue
|
||||
if ignore and any(pattern.match(name) for pattern in ignore):
|
||||
continue
|
||||
if args.unused and definition.get("used"):
|
||||
continue
|
||||
result[name] = definition
|
||||
|
||||
referrer_depth = args.referrers
|
||||
referrers = set()
|
||||
while referrer_depth:
|
||||
referrer_depth -= 1
|
||||
for entry in result.values():
|
||||
for used_by in entry.get("used", ()):
|
||||
referrers.add(used_by)
|
||||
for name, definition in names.items():
|
||||
if name not in referrers:
|
||||
continue
|
||||
if ignore and any(pattern.match(name) for pattern in ignore):
|
||||
continue
|
||||
result[name] = definition
|
||||
|
||||
referred_depth = args.referred
|
||||
referred = set()
|
||||
while referred_depth:
|
||||
referred_depth -= 1
|
||||
for entry in result.values():
|
||||
for uses in entry.get("uses", ()):
|
||||
referred.add(uses)
|
||||
for name, definition in names.items():
|
||||
if name not in referred:
|
||||
continue
|
||||
if ignore and any(pattern.match(name) for pattern in ignore):
|
||||
continue
|
||||
result[name] = definition
|
||||
|
||||
if args.format == "yaml":
|
||||
yaml.dump(result, sys.stdout, default_flow_style=False)
|
||||
elif args.format == "dot":
|
||||
print("digraph {")
|
||||
for name, entry in result.items():
|
||||
print(name)
|
||||
for used_by in entry.get("used", ()):
|
||||
if used_by in result:
|
||||
print(used_by, "->", name)
|
||||
print("}")
|
||||
else:
|
||||
raise ValueError("Unknown format %r" % (args.format))
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user