This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.
This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
Split admin API for reported events in detail und list view.
API was introduced with #8217 in synapse v.1.21.0.
It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview.
The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`.
It is similar to room and users API.
It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec).
Signed-off-by: Dirk Klimpel dirk@klimpel.org
* commit 'cc40a59b4':
1.21.0
Add prometheus metrics to track federation delays (#8430)
Make token serializing/deserializing async (#8427)
Allow additional SSO properties to be passed to the client (#8413)
changelog
Add an improved "forward extremities" metric
Rewrite BucketCollector
Fix _exposition.py to stop stripping samples
Drop support for ancient prometheus_client (#8426)
Various clean ups to room stream tokens. (#8423)
changelog
Report state res metrics to Prometheus and log
Move Measure calls into `resolve_events_with_store`
Expose a `get_resource_usage` method in `Measure`
Move `resolve_events_with_store` into StateResolutionHandler
* commit 'f43c66d23':
Add support for running Complement against the local checkout (#8317)
Filter out appservices from mau count (#8404)
Only assert valid next_link params when provided (#8417)
Add metrics to track success/otherwise of replication requests (#8406)
Fix handling of connection timeouts in outgoing http requests (#8400)
Changelog
Don't check whether a 3pid is allowed to register during password reset
Add checks for postgres sequence consistency (#8402)
Create a mechanism for marking tests "logcontext clean" (#8399)
Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410)
A pair of tiny cleanups in the federation request code. (#8401)
typo
* commit '31acc5c30':
Escape the error description on the sso_error template. (#8405)
Fix occasional "Re-starting finished log context" from keyring (#8398)
Allow existing users to login via OpenID Connect. (#8345)
Fix schema delta for servers that have not backfilled (#8396)
Fix MultiWriteIdGenerator's handling of restarts. (#8374)
s/URLs/variables in changelog
s/accidentally/incorrectly in changelog
Update changelog wording
Add type annotations to SimpleHttpClient (#8372)
Add new sequences to port DB script (#8387)
Add EventStreamPosition type (#8388)
Mark the shadow_banned column as boolean in synapse_port_db. (#8386)
* commit '2983049a7':
Factor out `_send_dummy_event_for_room` (#8370)
Improve logging of state resolution (#8371)
Fix bug which caused failure on join with malformed membership events (#8385)
Use `async with` for ID gens (#8383)
Don't push if an user account has expired (#8353)
Do not check lint/test dependencies at runtime. (#8377)
Add note to reverse_proxy.md about disabling Apache's mod_security2 (#8375)
Changelog
* commit '4325be1a5':
Fix missing null character check on guest_access room state
Fixed a bug with reactivating users with the admin API (#8362)
Admin API for reported events (#8217)
Fix wording of deprecation notice in changelog
Deprecation warning for synapse admin api being accessible under /_matrix
Create function to check for long names in devices (#8364)
Add a comment re #1691
Fix a bad merge from release-v1.20.0. (#8354)
Admin API for querying rooms where a user is a member (#8306)
Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup (#8322)
Simplify super() calls to Python 3 syntax. (#8344)
Allow appservice users to /login (#8320)
Update test logging to be able to accept braces (#8335)
Move lint dependencies to extras_require (#8330)
* commit '837293c31':
Remove obsolete __future__ imports (#8337)
Use admin_patterns for all admin APIs. (#8331)
Fix a potential bug of UnboundLocalError (#8329)
Switch metaclass initialization to python 3-compatible syntax (#8326)
Catch-up after Federation Outage (split, 4): catch-up loop (#8272)
Use slots in attrs classes where possible (#8296)
Fix typos in comments.
Add the topic and avatar to the room details admin API (#8305)
Improve SAML error messages (#8248)
Add experimental support for sharding event persister. Again. (#8294)
Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)
Use TLSv1.2 for fake servers in tests (#8208)
Add /_synapse/client to the reverse proxy docs (#8227)
Clean up `Notifier.on_new_room_event` code path (#8288)
* commit '2832ef5bb':
1.20.0rc3
Fix the exception that is raised when invalid JSON is encountered. (#8291)
Remove shared rooms info from upgrade/workers doc as it's still experimental (#8290)
* commit 'a3a90ee03':
Show a confirmation page during user password reset (#8004)
Do not error when thumbnailing invalid files (#8236)
Remove some unused distributor signals (#8216)
Fixup pusher pool notifications (#8287)
Revert "Fixup pusher pool notifications"
Fixup pusher pool notifications
* commit '525efab61':
1.20.0rc1
Systemd docs: configure workers to start after main process. (#8276)
Only add rows to the push actions table if the event notifies or should be marked unread (#8274)
Avoid table-scanning users at startup (#8271)
Fix unread count failing on NULL values (#8270)
Add more logging to debug slow startup (#8264)
Refuse to upgrade database on worker processes (#8266)
* commit '0d4f614fd':
Refactor `_get_e2e_device_keys_for_federation_query_txn` (#8225)
Add experimental support for sharding event persister. (#8170)
Add /user/{user_id}/shared_rooms/ api (#7785)
Do not try to store invalid data in the stats table (#8226)
Convert the main methods run by the reactor to async. (#8213)
* commit 'abeab964d':
Make _get_e2e_device_keys_and_signatures_txn return an attrs (#8224)
Fix errors when updating the user directory with invalid data (#8223)
Explain better what GDPR-erased means (#8189)
Convert additional databases to async/await part 3 (#8201)
Convert appservice code to async/await. (#8207)
Rename `_get_e2e_device_keys_txn` (#8222)
* commit '56efa9ec7': (22 commits)
Fix rate limiting unit tests. (#8167)
Add functions to `MultiWriterIdGen` used by events stream (#8164)
Do not allow send_nonmember_event to be called with shadow-banned users. (#8158)
Changelog fixes
Make StreamIdGen `get_next` and `get_next_mult` async (#8161)
Wording fixes to 'name' user admin api filter (#8163)
Fix missing double-backtick in RST document
Search in columns 'name' and 'displayname' in the admin users endpoint (#7377)
Add type hints for state. (#8140)
Stop shadow-banned users from sending non-member events. (#8142)
Allow capping a room's retention policy (#8104)
Add healthcheck for default localhost 8008 port on /health endpoint. (#8147)
Fix flaky shadow-ban tests. (#8152)
Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991)
Do not apply ratelimiting on joins to appservices (#8139)
Micro-optimisations to get_auth_chain_ids (#8132)
Allow denying or shadow banning registrations via the spam checker (#8034)
Stop shadow-banned users from sending invites. (#8095)
Be more tolerant of membership events in unknown rooms (#8110)
Improve the error code when trying to register using a name reserved for guests. (#8135)
...
* commit 'c9c544cda':
Remove `ChainedIdGenerator`. (#8123)
Switch the JSON byte producer from a pull to a push producer. (#8116)
Updated docs: Added note about missing 308 redirect support. (#8120)
Be stricter about JSON that is accepted by Synapse (#8106)
Convert runWithConnection to async. (#8121)
Remove the unused inlineCallbacks code-paths in the caching code (#8119)
Separate `get_current_token` into two. (#8113)
Convert events worker database to async/await. (#8071)
Add a link to the matrix-synapse-rest-password-provider. (#8111)
* commit 'e04e465b4':
Use the default templates when a custom template file cannot be found (#8037)
Changelog changes
Convert stream database to async/await. (#8074)
Add a shadow-banned flag to users. (#8092)
Convert pusher databases to async/await. (#8075)
Convert receipts and events databases to async/await. (#8076)
* commit 'db131b6b2':
Change the default log config to reduce disk I/O and storage (#8040)
Implement login blocking based on SAML attributes (#8052)
Add an assertion on prev_events in create_new_client_event (#8041)
Typo
Lint
why mypy why
Lint
Incorporate review
Incorporate review
Fix PUT /pushrules to use the right rule IDs
Back out the database hack and replace it with a temporary config setting
Fix cache name
Fix cache invalidation calls
Lint
Changelog
Implement new experimental push rules with a database hack to enable them
* commit '3950ae51e':
Ensure that remove_pusher is always async (#7981)
Ensure the msg property of HttpResponseException is a string. (#7979)
Remove from the event_relations table when purging historical events. (#7978)
Add additional logging for SAML sessions. (#7971)
Add MSC reference to changelog for #7736
Re-implement unread counts (#7736)
Various improvements to the docs (#7899)
Convert storage layer to async/await. (#7963)
Add an option to disable purge in delete room admin API (#7964)
Move some log lines from default logger to sql/transaction loggers (#7952)
Use the JSON module from the std library instead of simplejson. (#7936)
Fix exit code for `check_line_terminators.sh` (#7970)
Option to allow server admins to join complex rooms (#7902)
Fix typo in metrics docs (#7966)
Add script for finding files with unix line terminators (#7965)
Convert the remaining media repo code to async / await. (#7947)
Convert a synapse.events to async/await. (#7949)
Convert groups and visibility code to async / await. (#7951)
Convert push to async/await. (#7948)
Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress.
This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
