1
0
Commit Graph

697 Commits

Author SHA1 Message Date
Andrew Morgan
8f05de1d97 Merge commit 'de5cafe98' into dinsic 2020-12-31 11:43:55 +00:00
Andrew Morgan
b30484ab34 Merge commit '3ee17585c' into dinsic 2020-12-31 11:23:56 +00:00
Andrew Morgan
5fc0dd8126 Merge commit '74976a8e4' into dinsic 2020-12-31 11:23:24 +00:00
Andrew Morgan
63e8ab5481 Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page (#74) 2020-12-30 17:43:08 +00:00
Patrick Cloke
0d86b11666 Support PyJWT v2.0.0. (#8986)
Tests were broken due to an API changing. The code used in Synapse
proper should be compatible with both versions already.
2020-12-29 18:41:36 +00:00
Andrew Morgan
0ebdde8739 Override any missing default power level keys with DINUM's defaults when creating a room (#68)
The createRoom flow in DINUM's Synapse (through the AccessRules module which has hooks for all of this) already rejects a power levels content dict if it doesn't have high enough power levels to satisfy DINUM's [requirements](ac50ed353b/synapse/third_party_rules/access_rules.py (L233-L252)).

This PR ensures that any keys that aren't provided are replaced with the defaults, instead of just assuming the whole dict was correct (and thus those keys were set to mainline Synapse's default instead).
2020-10-29 10:48:29 +00:00
Andrew Morgan
854764073c Merge commit 'cc40a59b4' into anoa/dinsic_release_1_21_x
* commit 'cc40a59b4':
  1.21.0
  Add prometheus metrics to track federation delays (#8430)
  Make token serializing/deserializing async (#8427)
  Allow additional SSO properties to be passed to the client (#8413)
  changelog
  Add an improved "forward extremities" metric
  Rewrite BucketCollector
  Fix _exposition.py to stop stripping samples
  Drop support for ancient prometheus_client (#8426)
  Various clean ups to room stream tokens. (#8423)
  changelog
  Report state res metrics to Prometheus and log
  Move Measure calls into `resolve_events_with_store`
  Expose a `get_resource_usage` method in `Measure`
  Move `resolve_events_with_store` into StateResolutionHandler
2020-10-21 14:53:23 +01:00
Andrew Morgan
ab26b5894a Merge commit 'f43c66d23' into anoa/dinsic_release_1_21_x
* commit 'f43c66d23':
  Add support for running Complement against the local checkout (#8317)
  Filter out appservices from mau count (#8404)
  Only assert valid next_link params when provided (#8417)
  Add metrics to track success/otherwise of replication requests (#8406)
  Fix handling of connection timeouts in outgoing http requests (#8400)
  Changelog
  Don't check whether a 3pid is allowed to register during password reset
  Add checks for postgres sequence consistency (#8402)
  Create a mechanism for marking tests "logcontext clean" (#8399)
  Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410)
  A pair of tiny cleanups in the federation request code. (#8401)
  typo
2020-10-21 14:53:14 +01:00
Andrew Morgan
e529fa7574 Merge commit '4325be1a5' into anoa/dinsic_release_1_21_x
* commit '4325be1a5':
  Fix missing null character check on guest_access room state
  Fixed a bug with reactivating users with the admin API (#8362)
  Admin API for reported events (#8217)
  Fix wording of deprecation notice in changelog
  Deprecation warning for synapse admin api being accessible under /_matrix
  Create function to check for long names in devices (#8364)
  Add a comment re #1691
  Fix a bad merge from release-v1.20.0. (#8354)
  Admin API for querying rooms where a user is a member (#8306)
  Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup (#8322)
  Simplify super() calls to Python 3 syntax. (#8344)
  Allow appservice users to /login (#8320)
  Update test logging to be able to accept braces (#8335)
  Move lint dependencies to extras_require (#8330)
2020-10-21 14:14:36 +01:00
Andrew Morgan
7176832124 Merge commit '837293c31' into anoa/dinsic_release_1_21_x
* commit '837293c31':
  Remove obsolete __future__ imports (#8337)
  Use admin_patterns for all admin APIs. (#8331)
  Fix a potential bug of UnboundLocalError (#8329)
  Switch metaclass initialization to python 3-compatible syntax (#8326)
  Catch-up after Federation Outage (split, 4): catch-up loop (#8272)
  Use slots in attrs classes where possible (#8296)
  Fix typos in comments.
  Add the topic and avatar to the room details admin API (#8305)
  Improve SAML error messages (#8248)
  Add experimental support for sharding event persister. Again. (#8294)
  Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)
  Use TLSv1.2 for fake servers in tests (#8208)
  Add /_synapse/client to the reverse proxy docs (#8227)
  Clean up `Notifier.on_new_room_event` code path (#8288)
2020-10-20 19:51:28 +01:00
Andrew Morgan
8ee3f063a8 Merge commit 'a3a90ee03' into anoa/dinsic_release_1_21_x
* commit 'a3a90ee03':
  Show a confirmation page during user password reset (#8004)
  Do not error when thumbnailing invalid files (#8236)
  Remove some unused distributor signals (#8216)
  Fixup pusher pool notifications (#8287)
  Revert "Fixup pusher pool notifications"
  Fixup pusher pool notifications
2020-10-20 19:27:41 +01:00
Andrew Morgan
45079599ab Merge commit '453dfe210' into anoa/dinsic_release_1_21_x
* commit '453dfe210':
  blacklist MSC2753 sytests until it's implemented in synapse (#8285)
  Don't remember `enabled` of deleted push rules and properly return 404 for missing push rules in `.../actions` and `.../enabled` (#7796)
2020-10-20 19:15:10 +01:00
Andrew Morgan
061b9895eb Merge commit 'cca03dbec' into anoa/dinsic_release_1_21_x
* commit 'cca03dbec':
  fix typo
  s/fixes/fix/
  Directly import json from the standard library. (#8259)
  Allow for make_awaitable's return value to be re-used. (#8261)
  Rename 'populate_stats_process_rooms_2' background job back to 'populate_stats_process_rooms' again (#8243)
2020-10-20 18:29:12 +01:00
Andrew Morgan
47d53b32e3 Merge commit '17fa4c7ca' into anoa/dinsic_release_1_21_x
* commit '17fa4c7ca':
  Catch up after Federation Outage (split, 2): Track last successful stream ordering after transmission (#8247)
  Catch-up after Federation Outage (split, 1) (#8230)
  Fix type signature in simple_select_one_onecol and friends (#8241)
  Stop sub-classing object (#8249)
2020-10-20 18:09:59 +01:00
Andrew Morgan
d7fd0deaf1 Merge commit '112266eaf' into anoa/dinsic_release_1_21_x
* commit '112266eaf':
  Add StreamStore to mypy (#8232)
  Re-implement unread counts (again) (#8059)
2020-10-20 17:51:52 +01:00
Andrew Morgan
56722c69c5 Merge commit '0d4f614fd' into anoa/dinsic_release_1_21_x
* commit '0d4f614fd':
  Refactor `_get_e2e_device_keys_for_federation_query_txn` (#8225)
  Add experimental support for sharding event persister. (#8170)
  Add /user/{user_id}/shared_rooms/ api (#7785)
  Do not try to store invalid data in the stats table (#8226)
  Convert the main methods run by the reactor to async. (#8213)
2020-10-20 17:51:35 +01:00
Andrew Morgan
ff8a9a6a28 Merge commit 'e00816ad9' into anoa/dinsic_release_1_21_x
* commit 'e00816ad9':
  Do not yield on awaitables in tests. (#8193)
2020-10-20 17:42:05 +01:00
Andrew Morgan
ed944cb079 Merge commit 'b49a5b930' into anoa/dinsic_release_1_21_x
* commit 'b49a5b930':
  Convert stats and related calls to async/await (#8192)
2020-10-20 17:42:00 +01:00
Andrew Morgan
df77da8b4d Merge commit 'a466b6797' into anoa/dinsic_release_1_21_x
* commit 'a466b6797':
  Reduce run-times of tests by advancing the reactor less (#7757)
  Update debian systemd service to use Type=notify (#8169)
  Remove remaining is_guest argument uses from get_room_data calls (#8181)
  Do not propagate typing notifications from shadow-banned users. (#8176)
  Remove unused parameter from, and add safeguard in, get_room_data (#8174)
  Add required Debian dependencies to allow docker builds on the arm platform (#8144)
  Allow running mypy directly. (#8175)
  Update the test federation client to handle streaming responses (#8130)
  Do not propagate profile changes of shadow-banned users into rooms. (#8157)
  Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator (#8171)
  Convert simple_select_one and simple_select_one_onecol to async (#8162)
2020-10-20 17:13:42 +01:00
Andrew Morgan
b7bb088b84 Merge commit '56efa9ec7' into anoa/dinsic_release_1_21_x
* commit '56efa9ec7': (22 commits)
  Fix rate limiting unit tests. (#8167)
  Add functions to `MultiWriterIdGen` used by events stream (#8164)
  Do not allow send_nonmember_event to be called with shadow-banned users. (#8158)
  Changelog fixes
  Make StreamIdGen `get_next` and `get_next_mult` async  (#8161)
  Wording fixes to 'name' user admin api filter (#8163)
  Fix missing double-backtick in RST document
  Search in columns 'name' and 'displayname' in the admin users endpoint (#7377)
  Add type hints for state. (#8140)
  Stop shadow-banned users from sending non-member events. (#8142)
  Allow capping a room's retention policy (#8104)
  Add healthcheck for default localhost 8008 port on /health endpoint. (#8147)
  Fix flaky shadow-ban tests. (#8152)
  Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991)
  Do not apply ratelimiting on joins to appservices (#8139)
  Micro-optimisations to get_auth_chain_ids (#8132)
  Allow denying or shadow banning registrations via the spam checker (#8034)
  Stop shadow-banned users from sending invites. (#8095)
  Be more tolerant of membership events in unknown rooms (#8110)
  Improve the error code when trying to register using a name reserved for guests. (#8135)
  ...
2020-10-20 16:45:58 +01:00
Andrew Morgan
385ab7a5c8 Merge commit 'b79d69796' into anoa/dinsic_release_1_21_x
* commit 'b79d69796':
  1.19.1rc1
  Fix join ratelimiter breaking profile updates and idempotency (#8153)
2020-10-20 16:21:14 +01:00
Andrew Morgan
74f29284aa Remove some extraneous @unittest.INFOs on unit tests (#8592) 2020-10-20 11:49:15 +01:00
Andrew Morgan
687d30b2ed Merge commit 'c9c544cda' into anoa/dinsic_release_1_21_x
* commit 'c9c544cda':
  Remove `ChainedIdGenerator`. (#8123)
  Switch the JSON byte producer from a pull to a push producer. (#8116)
  Updated docs: Added note about missing 308 redirect support. (#8120)
  Be stricter about JSON that is accepted by Synapse (#8106)
  Convert runWithConnection to async. (#8121)
  Remove the unused inlineCallbacks code-paths in the caching code (#8119)
  Separate `get_current_token` into two. (#8113)
  Convert events worker database to async/await. (#8071)
  Add a link to the matrix-synapse-rest-password-provider. (#8111)
2020-10-19 19:12:39 +01:00
Andrew Morgan
20bfb5390c Merge commit '5dd73d029' into anoa/dinsic_release_1_21_x
* commit '5dd73d029':
  Add type hints to handlers.message and events.builder (#8067)
2020-10-19 18:24:25 +01:00
Andrew Morgan
9dd3b52f1b Merge commit '7620912d8' into anoa/dinsic_release_1_21_x
* commit '7620912d8':
  Add health check endpoint (#8048)
2020-10-19 17:06:32 +01:00
Andrew Morgan
d47db9e9e9 Merge commit '2ffd6783c' into anoa/dinsic_release_1_21_x
* commit '2ffd6783c':
  Revert #7736 (#8039)
2020-10-19 17:06:14 +01:00
Andrew Morgan
383a87e0e2 Merge commit 'd4a7829b1' into anoa/dinsic_release_1_21_x
* commit 'd4a7829b1':
  Convert synapse.api to async/await (#8031)
2020-10-19 14:59:45 +01:00
Andrew Morgan
9d09ab74bb Merge commit '66f24449d' into anoa/dinsic_release_1_21_x
* commit '66f24449d':
  Improve performance of the register endpoint (#8009)
2020-10-19 14:29:04 +01:00
Andrew Morgan
d174faacc7 Merge commit 'a7bdf98d0' into anoa/dinsic_release_1_21_x
* commit 'a7bdf98d0':
  Rename database classes to make some sense (#8033)
2020-10-16 18:21:05 +01:00
Andrew Morgan
73381abc23 Merge commit '5d92a1428' into anoa/dinsic_release_1_21_x
* commit '5d92a1428':
  Prevent join->join membership transitions changing member count (#7977)
2020-10-16 17:21:08 +01:00
Andrew Morgan
4ae35a3c7b Merge commit 'c978f6c45' into anoa/dinsic_release_1_21_x
* commit 'c978f6c45':
  Convert federation client to async/await. (#7975)
2020-10-16 17:08:04 +01:00
Andrew Morgan
684991eff0 Merge commit '3950ae51e' into anoa/dinsic_release_1_21_x
* commit '3950ae51e':
  Ensure that remove_pusher is always async (#7981)
  Ensure the msg property of HttpResponseException is a string. (#7979)
  Remove from the event_relations table when purging historical events. (#7978)
  Add additional logging for SAML sessions. (#7971)
  Add MSC reference to changelog for #7736
  Re-implement unread counts (#7736)
  Various improvements to the docs (#7899)
  Convert storage layer to async/await. (#7963)
  Add an option to disable purge in delete room admin API (#7964)
  Move some log lines from default logger to sql/transaction loggers (#7952)
  Use the JSON module from the std library instead of simplejson. (#7936)
  Fix exit code for `check_line_terminators.sh` (#7970)
  Option to allow server admins to join complex rooms (#7902)
  Fix typo in metrics docs (#7966)
  Add script for finding files with unix line terminators (#7965)
  Convert the remaining media repo code to async / await. (#7947)
  Convert a synapse.events to async/await. (#7949)
  Convert groups and visibility code to async / await. (#7951)
  Convert push to async/await. (#7948)
2020-10-16 17:07:28 +01:00
Richard van der Hoff
898196f1cc guard against accidental modification 2020-10-13 23:24:50 +01:00
Richard van der Hoff
617e8a4653 Allow ThirdPartyRules modules to replace event content
Support returning a new event dict from `check_event_allowed`.
2020-10-13 23:24:50 +01:00
Andrew Morgan
722e1c016a "Freeze" a room when the last admin of that room leaves (#59)
If the last admin of a room departs, and thus the room no longer has any admins within it, we "freeze" the room. Freezing a room means that the power level required to do anything in the room (sending messages, inviting others etc) will require power level 100.

At the moment, an admin can come back and unfreeze the room manually. The plan is to eventually make unfreezing of the room automatic on admin rejoin, though that will be in a separate PR.

This *could* work in mainline, however if the admin who leaves is on a homeserver without this functionality, then the room isn't frozen. I imagine this would probably be pretty confusing to people. Part of this feature was allowing Synapse modules to send events, which has been implemented in mainline at  https://github.com/matrix-org/synapse/pull/8479, and cherry-picked to the `dinsic` fork in 62c7b10. The actual freezing logic has been implemented here in the RoomAccessRules module.
2020-10-13 15:49:50 +01:00
Erik Johnston
b2486f6656 Fix message duplication if something goes wrong after persisting the event (#8476)
Should fix #3365.
2020-10-13 12:07:56 +01:00
Andrew Morgan
62c7b10ea5 Allow modules to create and send events into rooms (#8479)
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.

The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.

This commit has been cherry-picked from mainline.
2020-10-12 13:24:15 +01:00
Andrew Morgan
66ac4b1e34 Allow modules to create and send events into rooms (#8479)
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.

The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.
2020-10-09 13:46:36 +01:00
Patrick Cloke
c9c0ad5e20 Remove the deprecated Handlers object (#8494)
All handlers now available via get_*_handler() methods on the HomeServer.
2020-10-09 07:24:34 -04:00
Richard van der Hoff
a024461130 Additional tests for third-party event rules (#8468)
* Optimise and test state fetching for 3p event rules

Getting all the events at once is much more efficient than getting them
individually

* Test that 3p event rules can modify events
2020-10-06 16:31:31 +01:00
Richard van der Hoff
9c0b168cff Merge pull request #8467 from matrix-org/rav/fix_3pevent_rules
Fix third-party event modules for `check_visibility_can_be_modified` check
2020-10-06 11:32:53 +01:00
Richard van der Hoff
785437dc0d Update default room version to 6 (#8461)
Per https://github.com/matrix-org/matrix-doc/pull/2788
2020-10-05 21:40:51 +01:00
Richard van der Hoff
4cd1448d0e Fix third-party event modules for check_visibility_can_be_modified check
PR #8292 tried to maintain backwards compat with modules which don't provide a
`check_visibility_can_be_modified` method, but the tests weren't being run,
and the check didn't work.
2020-10-05 20:29:52 +01:00
Andrew Morgan
0991a2da93 Allow ThirdPartyEventRules modules to manipulate public room state (#8292)
This PR allows `ThirdPartyEventRules` modules to view, manipulate and block changes to the state of whether a room is published in the public rooms directory.

While the idea of whether a room is in the public rooms list is not kept within an event in the room, `ThirdPartyEventRules` generally deal with controlling which modifications can happen to a room. Public rooms fits within that idea, even if its toggle state isn't controlled through a state event.
2020-10-05 14:57:46 +01:00
Erik Johnston
7941372ec8 Make token serializing/deserializing async (#8427)
The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
2020-09-30 20:29:19 +01:00
Erik Johnston
ea70f1c362 Various clean ups to room stream tokens. (#8423) 2020-09-29 21:48:33 +01:00
Andrew Morgan
1c6b8752b8 Only assert valid next_link params when provided (#8417)
Broken in https://github.com/matrix-org/synapse/pull/8275 and has yet to be put in a release. Fixes https://github.com/matrix-org/synapse/issues/8418.

`next_link` is an optional parameter. However, we were checking whether the `next_link` param was valid, even if it wasn't provided. In that case, `next_link` was `None`, which would clearly not be a valid URL.

This would prevent password reset and other operations if `next_link` was not provided, and the `next_link_domain_whitelist` config option was set.
2020-09-29 12:36:44 +01:00
Dirk Klimpel
8998217540 Fixed a bug with reactivating users with the admin API (#8362)
Fixes: #8359 

Trying to reactivate a user with the admin API (`PUT /_synapse/admin/v2/users/<user_name>`) causes an internal server error.

Seems to be a regression in #8033.
2020-09-22 18:19:01 +01:00
Dirk Klimpel
4da01f9c61 Admin API for reported events (#8217)
Add an admin API to read entries of table `event_reports`. API: `GET /_synapse/admin/v1/event_reports`
2020-09-22 18:15:04 +01:00
Dionysis Grigoropoulos
37ca5924bd Create function to check for long names in devices (#8364)
* Create a new function to verify that the length of a device name is
under a certain threshold.
* Refactor old code and tests to use said function.
* Verify device name length during registration of device
* Add a test for the above

Signed-off-by: Dionysis Grigoropoulos <dgrig@erethon.com>
2020-09-22 11:42:55 +01:00