1
0
Commit Graph

634 Commits

Author SHA1 Message Date
Andrew Morgan
687d30b2ed Merge commit 'c9c544cda' into anoa/dinsic_release_1_21_x
* commit 'c9c544cda':
  Remove `ChainedIdGenerator`. (#8123)
  Switch the JSON byte producer from a pull to a push producer. (#8116)
  Updated docs: Added note about missing 308 redirect support. (#8120)
  Be stricter about JSON that is accepted by Synapse (#8106)
  Convert runWithConnection to async. (#8121)
  Remove the unused inlineCallbacks code-paths in the caching code (#8119)
  Separate `get_current_token` into two. (#8113)
  Convert events worker database to async/await. (#8071)
  Add a link to the matrix-synapse-rest-password-provider. (#8111)
2020-10-19 19:12:39 +01:00
Andrew Morgan
20bfb5390c Merge commit '5dd73d029' into anoa/dinsic_release_1_21_x
* commit '5dd73d029':
  Add type hints to handlers.message and events.builder (#8067)
2020-10-19 18:24:25 +01:00
Andrew Morgan
9dd3b52f1b Merge commit '7620912d8' into anoa/dinsic_release_1_21_x
* commit '7620912d8':
  Add health check endpoint (#8048)
2020-10-19 17:06:32 +01:00
Andrew Morgan
d47db9e9e9 Merge commit '2ffd6783c' into anoa/dinsic_release_1_21_x
* commit '2ffd6783c':
  Revert #7736 (#8039)
2020-10-19 17:06:14 +01:00
Andrew Morgan
383a87e0e2 Merge commit 'd4a7829b1' into anoa/dinsic_release_1_21_x
* commit 'd4a7829b1':
  Convert synapse.api to async/await (#8031)
2020-10-19 14:59:45 +01:00
Andrew Morgan
9d09ab74bb Merge commit '66f24449d' into anoa/dinsic_release_1_21_x
* commit '66f24449d':
  Improve performance of the register endpoint (#8009)
2020-10-19 14:29:04 +01:00
Andrew Morgan
d174faacc7 Merge commit 'a7bdf98d0' into anoa/dinsic_release_1_21_x
* commit 'a7bdf98d0':
  Rename database classes to make some sense (#8033)
2020-10-16 18:21:05 +01:00
Andrew Morgan
73381abc23 Merge commit '5d92a1428' into anoa/dinsic_release_1_21_x
* commit '5d92a1428':
  Prevent join->join membership transitions changing member count (#7977)
2020-10-16 17:21:08 +01:00
Andrew Morgan
4ae35a3c7b Merge commit 'c978f6c45' into anoa/dinsic_release_1_21_x
* commit 'c978f6c45':
  Convert federation client to async/await. (#7975)
2020-10-16 17:08:04 +01:00
Andrew Morgan
684991eff0 Merge commit '3950ae51e' into anoa/dinsic_release_1_21_x
* commit '3950ae51e':
  Ensure that remove_pusher is always async (#7981)
  Ensure the msg property of HttpResponseException is a string. (#7979)
  Remove from the event_relations table when purging historical events. (#7978)
  Add additional logging for SAML sessions. (#7971)
  Add MSC reference to changelog for #7736
  Re-implement unread counts (#7736)
  Various improvements to the docs (#7899)
  Convert storage layer to async/await. (#7963)
  Add an option to disable purge in delete room admin API (#7964)
  Move some log lines from default logger to sql/transaction loggers (#7952)
  Use the JSON module from the std library instead of simplejson. (#7936)
  Fix exit code for `check_line_terminators.sh` (#7970)
  Option to allow server admins to join complex rooms (#7902)
  Fix typo in metrics docs (#7966)
  Add script for finding files with unix line terminators (#7965)
  Convert the remaining media repo code to async / await. (#7947)
  Convert a synapse.events to async/await. (#7949)
  Convert groups and visibility code to async / await. (#7951)
  Convert push to async/await. (#7948)
2020-10-16 17:07:28 +01:00
Andrew Morgan
722e1c016a "Freeze" a room when the last admin of that room leaves (#59)
If the last admin of a room departs, and thus the room no longer has any admins within it, we "freeze" the room. Freezing a room means that the power level required to do anything in the room (sending messages, inviting others etc) will require power level 100.

At the moment, an admin can come back and unfreeze the room manually. The plan is to eventually make unfreezing of the room automatic on admin rejoin, though that will be in a separate PR.

This *could* work in mainline, however if the admin who leaves is on a homeserver without this functionality, then the room isn't frozen. I imagine this would probably be pretty confusing to people. Part of this feature was allowing Synapse modules to send events, which has been implemented in mainline at  https://github.com/matrix-org/synapse/pull/8479, and cherry-picked to the `dinsic` fork in 62c7b10. The actual freezing logic has been implemented here in the RoomAccessRules module.
2020-10-13 15:49:50 +01:00
Andrew Morgan
62c7b10ea5 Allow modules to create and send events into rooms (#8479)
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.

The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.

This commit has been cherry-picked from mainline.
2020-10-12 13:24:15 +01:00
Andrew Morgan
8b234fb216 Swap method calls in RoomAccessTestCase.test_change_rules (#64)
Swap these calls around, as the check for changing `restricted` to `unrestricted` will actually change `self.restricted_room` to an unrestricted room.

Do that last, instead of first. Additionally add a comment with a warning.
2020-09-18 11:37:21 +01:00
Andrew Morgan
3fe1c8485b Make AccessRules use the public rooms directory instead of checking a room's join rules on rule change (#63)
This PR switches several conditions regarding room access rules to check against the status of the room's inclusion in the public room list instead of its join rules.

The code includes a snapshot of https://github.com/matrix-org/synapse/pull/8292, which will likely change in time and need merging in again.
2020-09-18 11:30:36 +01:00
Andrew Morgan
319d07373a Override the power levels defaults, enforce mod requirement for invites, admin requirements for unknown state events (#61)
This PR modifies the `RoomAccessRules` module, an implementation of `ThirdPartyEventRules`, to both:

* Modify the default power levels when creating a room to set:
  - `invite` to be minimum PL50
  - `state_default` to be minimum PL100
* Enforce this when creating the room.
2020-09-11 15:47:09 +01:00
Andrew Morgan
3d1c941a6e RoomAccessRules cleanup (#62)
Various cleanups of the DINUM-specific `RoomAccessRules` module, including:

* Type hints
* Docstring cleanups
* Some code cleanups
2020-09-10 19:04:34 +01:00
Andrew Morgan
ff91a451b1 Add a config option for validating 'next_link' parameters against a domain whitelist (#8275)
This is a config option ported over from DINUM's Sydent: https://github.com/matrix-org/sydent/pull/285

They've switched to validating 3PIDs via Synapse rather than Sydent, and would like to retain this functionality.

This original purpose for this change is phishing prevention. This solution could also potentially be replaced by a similar one to https://github.com/matrix-org/synapse/pull/8004, but across all `*/submit_token` endpoint.

This option may still be useful to enterprise even with that safeguard in place though, if they want to be absolutely sure that their employees don't follow links to other domains.
2020-09-09 13:31:25 +01:00
Patrick Cloke
f594e434c3 Switch the JSON byte producer from a pull to a push producer. (#8116) 2020-08-19 08:07:57 -04:00
Erik Johnston
5dd73d029e Add type hints to handlers.message and events.builder (#8067) 2020-08-12 15:05:50 +01:00
Erik Johnston
7620912d84 Add health check endpoint (#8048) 2020-08-07 14:21:24 +01:00
Brendan Abolivier
2ffd6783c7 Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke
d4a7829b12 Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Patrick Cloke
66f24449dd Improve performance of the register endpoint (#8009) 2020-08-06 08:09:55 -04:00
Erik Johnston
a7bdf98d01 Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Andrew Morgan
7f2e76bef9 Merge commit 'f88c48f3b' into anoa/dinsic_release_1_18_x
* commit 'f88c48f3b':
  1.18.0rc1
  Fix error reporting when using `opentracing.trace` (#7961)
  Fix typing replication not being handled on master (#7959)
  Remove hacky error handling for inlineDeferreds. (#7950)
  Convert tests/rest/admin/test_room.py to unix file endings (#7953)
  Support oEmbed for media previews. (#7920)
  Convert state resolution to async/await (#7942)
  Fix up types and comments that refer to Deferreds. (#7945)
  Do not convert async functions to Deferreds in the interactive_auth_handler (#7944)
  Convert more of the media code to async/await (#7873)
  Return an empty body for OPTIONS requests. (#7886)
  Downgrade warning on client disconnect to INFO (#7928)
  Convert presence handler helpers to async/await. (#7939)
  Update the auth providers to be async. (#7935)
  Put a cache on `/state_ids` (#7931)
2020-08-03 17:51:46 -07:00
Andrew Morgan
a567e763ea Merge commit 'de119063f' into anoa/dinsic_release_1_18_x
* commit 'de119063f': (31 commits)
  Convert room list handler to async/await. (#7912)
  Element CSS and logo in email templates (#7919)
  Lint the contrib/ directory in CI and linting scripts, add synctl to linting script (#7914)
  Remove unused code from synapse.logging.utils. (#7897)
  Fix a typo in the sample config. (#7890)
  Fix deprecation warning: import ABC from collections.abc (#7892)
  Change sample config's postgres user to synapse_user (#7889)
  Fix deprecation warning due to invalid escape sequences (#7895)
  Remove Ubuntu Eoan that is now EOL (#7888)
  Fix the trace function for async functions. (#7872)
  Add help for creating a user via docker (#7885)
  Switch to Debian:Slim from Alpine for the docker image (#7839)
  Stop using 'device_max_stream_id' (#7882)
  Fix TypeError in synapse.notifier (#7880)
  Add a default limit (of 100) to get/sync operations. (#7858)
  Change "unknown room ver" logging to warning. (#7881)
  Convert device handler to async/await (#7871)
  Convert synapse.app to async/await. (#7868)
  Convert _base, profile, and _receipts handlers to async/await (#7860)
  Add admin endpoint to get members in a room. (#7842)
  ...
2020-08-03 17:38:45 -07:00
Andrew Morgan
d9e25be896 Merge commit 'a973bcb8a' into anoa/dinsic_release_1_18_x
* commit 'a973bcb8a':
  Add some tiny type annotations (#7870)
  Remove obsolete comment.
  Ensure that calls to `json.dumps` are compatible with the standard library json. (#7836)
  Avoid brand new rooms in `delete_old_current_state_events` (#7854)
  Allow accounts to be re-activated from the admin APIs. (#7847)
  Fix tests
  Fix typo
  Newsfile
  Use get_users_in_room rather than state handler in typing for speed
  Fix client reader sharding tests (#7853)
  Convert E2E key and room key handlers to async/await. (#7851)
  Return the proper 403 Forbidden error during errors with JWT logins. (#7844)
  remove `retry_on_integrity_error` wrapper for persist_events (#7848)
2020-08-03 17:31:36 -07:00
Andrew Morgan
16a44cff1d Merge commit '491f0dab1' into anoa/dinsic_release_1_18_x
* commit '491f0dab1':
  Add delete room admin endpoint (#7613)
2020-08-03 17:31:18 -07:00
Andrew Morgan
61ac6e22b9 Merge commit '77d2c0541' into anoa/dinsic_release_1_18_x
* commit '77d2c0541':
  Add the option to validate the `iss` and `aud` claims for JWT logins. (#7827)
2020-08-03 17:31:14 -07:00
Andrew Morgan
17cbac54f4 Merge commit '66a4af8d9' into anoa/dinsic_release_1_18_x
* commit '66a4af8d9':
  Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)
2020-08-03 17:29:38 -07:00
Andrew Morgan
530c8a5cda Merge commit '43726783e' into dinsic
* commit '43726783e': (22 commits)
  1.17.0rc1
  Fix some spelling mistakes / typos. (#7811)
  `update_membership` declaration: now always returns an event id. (#7809)
  Improve stacktraces from exceptions in background processes (#7808)
  Fix `can only concatenate list (not "tuple") to list` exception (#7810)
  Pass original request headers from workers to the main process. (#7797)
  Generate real events when we reject invites (#7804)
  Add `HomeServer.signing_key` property (#7805)
  Revert "Update the installation docs on apt-transport-https (#7801)"
  Do not use simplejson in Synapse. (#7800)
  Stop passing bytes when dumping JSON (#7799)
  Update the installation docs on apt-transport-https (#7801)
  shuffle changelog slightly
  Change Caddy links (old is deprecated) (#7789)
  Stop populating unused table `local_invites`. (#7793)
  Refactor getting replication updates from database v2. (#7740)
  Add libwebp dependency to Dockerfile (#7791)
  Add documentation for JWT login type and improve sample config. (#7776)
  Convert the appservice handler to async/await. (#7775)
  Don't ignore `set_tweak` actions with no explicit `value`. (#7766)
  ...
2020-08-03 17:19:26 -07:00
Andrew Morgan
60a96e4232 Merge commit '5cdca53aa' into dinsic
* commit '5cdca53aa':
  Merge different Resource implementation classes (#7732)
  Fix inconsistent handling of upper and lower cases of email addresses. (#7021)
  Allow YAML config file to contain None (#7779)
  Fix a typo.
  Move 1.15.2 after 1.16.0rc2.
  1.16.0rc2
  Remove an extraneous space.
  Add links to the fixes.
  Fix tense in the release notes.
  Hack to add push priority to push notifications (#7765)
  Add early returns to `_check_for_soft_fail` (#7769)
  Use symbolic names for replication stream names (#7768)
  Type checking for `FederationHandler` (#7770)
  Fix new metric where we used ms instead of seconds (#7771)
  Fix incorrect error message when database CTYPE was set incorrectly. (#7760)
  Pin link in CHANGES.md
  Fixes to CHANGES.md
2020-08-03 17:08:31 -07:00
Andrew Morgan
87f941c3fd Merge commit 'dc80a0762' into dinsic
* commit 'dc80a0762':
  1.16.0rc1
  Back out MSC2625 implementation (#7761)
  Additional configuration options for auto-join rooms (#7763)
  Add some metrics for inbound and outbound federation processing times (#7755)
  Explain the purpose of the "tests" conditional dependency requirement (#7751)
  Add another yield point to state res v2 (#7746)
  Move flake8 to end. Don't exit script on failure (#7738)
  Make tox actions work on Debian 10 (#7703)
  Yield during large v2 state res. (#7735)
  add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675)
  Set Content-Length for Metrics requests (#7730)
  Sync ignored table names in synapse_port_db to current database schema (#7717)
  Allow local media to be marked as safe from being quarantined. (#7718)
  Convert directory handler to async/await (#7727)
  Speed up state res v2 across large state differences. (#7725)
2020-08-03 16:29:02 -07:00
Andrew Morgan
3261eb71a6 Merge commit '434716e1d' into dinsic
* commit '434716e1d':
  Fetch from the r0 media path instead of the unspecced v1. (#7714)
2020-08-03 16:22:52 -07:00
Andrew Morgan
48e7f210c6 Merge commit 'a3f11567d' into dinsic
* commit 'a3f11567d':
  Replace all remaining six usage with native Python 3 equivalents (#7704)
2020-08-03 16:15:34 -07:00
Andrew Morgan
5d92a1428c Prevent join->join membership transitions changing member count (#7977)
`StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc.

However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information.

This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point.

Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update).

Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338.
2020-08-03 21:54:24 +01:00
Patrick Cloke
c978f6c451 Convert federation client to async/await. (#7975) 2020-07-30 08:01:33 -04:00
Brendan Abolivier
8dff4a1242 Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
Dirk Klimpel
e866e3b896 Add an option to disable purge in delete room admin API (#7964)
Add option ```purge``` to ```POST /_synapse/admin/v1/rooms/<room_id>/delete```
Fixes: #3761

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-28 20:08:23 +01:00
Andrew Morgan
c4268e3da6 Convert tests/rest/admin/test_room.py to unix file endings (#7953)
Converts tests/rest/admin/test_room.py to have unix file endings after they were accidentally changed in #7613.

Keeping the same changelog as #7613 as it hasn't gone out in a release yet.
2020-07-27 13:22:52 +01:00
Patrick Cloke
3fc8fdd150 Support oEmbed for media previews. (#7920)
Fixes previews of Twitter URLs by using their oEmbed endpoint to grab content.
2020-07-27 07:50:44 -04:00
Patrick Cloke
5ea29d7f85 Convert more of the media code to async/await (#7873) 2020-07-24 09:39:02 -04:00
Michael Albert
fff483ea96 Add admin endpoint to get members in a room. (#7842) 2020-07-16 16:43:23 -04:00
Patrick Cloke
8c7d0f163d Allow accounts to be re-activated from the admin APIs. (#7847) 2020-07-15 11:00:21 -04:00
Patrick Cloke
111e70d75c Return the proper 403 Forbidden error during errors with JWT logins. (#7844) 2020-07-15 07:10:21 -04:00
Dirk Klimpel
491f0dab1b Add delete room admin endpoint (#7613)
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
Patrick Cloke
77d2c05410 Add the option to validate the iss and aud claims for JWT logins. (#7827) 2020-07-14 07:16:43 -04:00
Patrick Cloke
66a4af8d96 Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802) 2020-07-10 14:30:08 -04:00
Patrick Cloke
38e1fac886 Fix some spelling mistakes / typos. (#7811) 2020-07-09 09:52:58 -04:00
Richard van der Hoff
76dbd7b8d6 Stop populating unused table local_invites. (#7793)
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
2020-07-07 14:20:40 +01:00