1
0

Compare commits

...

219 Commits

Author SHA1 Message Date
Andrew Morgan f6ea4f5b45 Replace deprecated --force dpkg option
While watching @Half-Shot install his new Synapse dpkg popped out an error that `--force` was deprecated.

This was on Ubuntu 19.04. Not sure if the `--force-all` option is supported by older distributions that we support
2019-10-19 21:17:20 +02:00
Andrew Morgan 22a9f75097 Delete format_tap.py (#6219)
* Delete format_tap.py

This python implementation of a tap formatting library for buildkite has been
replaced with a perl implementation as part of the matrix-org/sytest repo,
which is specific to sytest's language, not that of any one homeserver's.
2019-10-19 20:42:10 +03:00
Richard van der Hoff 93eaeec75a Remove Auth.check method (#6217)
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Hubert Chathi 0a9d22808c Merge branch 'uhoreg/e2e_cross-signing_merged' into develop 2019-10-18 15:57:40 +01:00
Richard van der Hoff 47ada4dffe Merge pull request #6214 from matrix-org/rav/event_auth/1
Remove a bunch of dead event_auth code.
2019-10-18 16:20:32 +02:00
Richard van der Hoff 2b22faded7 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 12:35:13 +01:00
Richard van der Hoff 560c122267 Fix logging config for the docker image (#6197)
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
2019-10-18 13:34:33 +02:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Hubert Chathi 770a6053a0 add note about database upgrade 2019-10-18 11:38:27 +01:00
Brendan Abolivier 1ee97cbd01 Merge branch 'master' into develop 2019-10-18 11:00:43 +01:00
Brendan Abolivier 9fc14b5e84 Merge tag 'v1.4.1'
Synapse 1.4.1 (2019-10-18)
==========================

No changes since 1.4.1rc1.

Synapse 1.4.1rc1 (2019-10-17)
=============================

Bugfixes
--------

- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](https://github.com/matrix-org/synapse/commit/5b0e9948eaae801643e594b5abc8ee4b10bd194e))
2019-10-18 10:59:01 +01:00
Hubert Chathi f0f6a2b360 use the right function for when we're already in runInteraction 2019-10-18 10:56:54 +01:00
Erik Johnston d98029ea89 Merge pull request #6196 from matrix-org/erikj/await
Move rest/admin to use async/await.
2019-10-18 11:53:02 +02:00
przemas75 c3772a71cb Update docker-compose.yml for a static config file, and update traefik examples (#6142) 2019-10-18 11:23:33 +02:00
Brendan Abolivier 774b8d0930 Fix changelog 2019-10-18 10:19:02 +01:00
Brendan Abolivier 41b9faed16 1.4.1 2019-10-18 10:15:12 +01:00
Bart Noordervliet dc4bec885d Add missing BOOLEAN_COLUMNs to synapse_port_db (#6216)
Small fix to synapse_port_db to be able to convert from database schema v56.
2019-10-18 11:13:59 +02:00
Richard van der Hoff 80003dfcd5 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 10:11:40 +01:00
Erik Johnston 5859a5c569 Fix presence timeouts when synchrotron restarts. (#6212)
* Fix presence timeouts when synchrotron restarts.

Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.

Fixes #3715.
2019-10-18 06:42:26 +01:00
Richard van der Hoff 1594de856c changelog 2019-10-17 21:45:59 +01:00
Richard van der Hoff 1ba359a11f rip out some unreachable code
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Richard van der Hoff 6fb0a3da07 Remove dead changelog file
This is part of 1.4.1
2019-10-17 18:03:28 +01:00
Richard van der Hoff 423f7ae397 Fix up changelogs 2019-10-17 17:06:07 +01:00
Richard van der Hoff 70d9759cdf Merge tag 'v1.4.1rc1' into develop
Synapse 1.4.1rc1 (2019-10-17)
=============================

Bugfixes
--------

- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](https://github.com/matrix-org/synapse/commit/5b0e9948eaae801643e594b5abc8ee4b10bd194e))
2019-10-17 17:03:02 +01:00
Richard van der Hoff 71cd3fed66 1.4.1rc1 2019-10-17 16:40:56 +01:00
Erik Johnston 5b0e9948ea Do the update as a background index 2019-10-17 16:31:42 +01:00
Erik Johnston a2bb50c2eb Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-17 16:29:07 +01:00
Hubert Chathi 06fc66c81e Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Hubert Chathi 132b251e29 expand on comment 2019-10-11 14:24:52 -04:00
Erik Johnston 2e97a4c197 Port synapse/rest/client/_base.py to async/await 2019-10-11 15:29:26 +01:00
Erik Johnston 3c2d6c708c Add maybe_awaitable and fix __init__ bugs 2019-10-11 15:26:09 +01:00
Hubert Chathi 691dd67fcd Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Erik Johnston 6f5c6c8f60 Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url
Trace non-JSON APIs, /media, /key etc
2019-10-11 13:23:52 +01:00
Valérian Rousset be9b55e0d2 cas: support setting display name (#6114)
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Erik Johnston c3b0fbe9c3 Newsfile 2019-10-11 12:24:52 +01:00
Erik Johnston dfbb62c28d Port synaps/rest/client/users.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston f95325e22a Port synaps/rest/client/server_notice_servlet.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston 281f887090 Port synaps/rest/client/media.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston fca3a541e7 Port rest/admin/__init__.py to async/await 2019-10-11 12:17:52 +01:00
Erik Johnston de3a176426 Newsfile 2019-10-11 11:58:52 +01:00
Erik Johnston f3ceaf4323 Trace non-JSON APIs, /media, /key etc 2019-10-11 11:58:52 +01:00
Richard van der Hoff cad0132fb5 Remove dead check_auth script
This doesn't work, and afaict hasn't been used since 2015.
2019-10-11 11:24:03 +01:00
Neil Johnson a0d0ba7862 Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi 7a0dce9259 make sure we actually return something 2019-10-10 20:31:30 -04:00
Hubert Chathi 2208891ace add changelog 2019-10-10 19:22:10 -04:00
Hubert Chathi 4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
Erik Johnston 83d86106a8 Merge pull request #6156 from matrix-org/erikj/postgres_any
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston b54b1e759a Fix SQLite take 2 2019-10-10 16:19:40 +01:00
Erik Johnston 329eae9cda Merge pull request #6186 from matrix-org/erikj/disable_sql_bytes
Disable bytes usage with postgres
2019-10-10 15:59:35 +01:00
Erik Johnston afb6d9d53b Fix SQLite 2019-10-10 15:55:41 +01:00
Erik Johnston bc244627ac Fix postgres unit tests 2019-10-10 15:37:53 +01:00
Erik Johnston 3bc687508f Remove add_in_list_sql_clause 2019-10-10 15:35:46 +01:00
Erik Johnston ca3e01e50d Fix store_url_cache using bytes 2019-10-10 14:52:29 +01:00
Erik Johnston 9d06fb9cb1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any 2019-10-10 14:38:21 +01:00
Erik Johnston 8bc529c04d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 14:30:51 +01:00
Erik Johnston 571872cf0f Merge pull request #6191 from matrix-org/rav/fix_packaging
Fix python packaging
2019-10-10 14:30:42 +01:00
Richard van der Hoff 430dc2c67b Fix python packaging
... after it got borked by #6081
2019-10-10 14:05:30 +01:00
Hubert Chathi 5373de6cce change test name to be unique 2019-10-10 08:54:07 -04:00
Erik Johnston 933034e2fe Merge pull request #6127 from matrix-org/erikj/patch_inner
Add more log context checks when patching inlineCallbacks
2019-10-10 13:47:50 +01:00
Erik Johnston fe1c1e6c28 Fixup comments
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-10 13:17:19 +01:00
Erik Johnston 203ccdac5f Newsfile 2019-10-10 13:15:49 +01:00
Erik Johnston b161786c14 Replace IN usage with helper funcs 2019-10-10 13:15:49 +01:00
Erik Johnston b4fbf71187 Add helper funcs to use postgres ANY
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston 91f43dca39 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 13:10:57 +01:00
werner291 b5b03b7079 Add domain validation when creating room with list of invitees (#6121) 2019-10-10 13:05:48 +01:00
Michael Telatynski 9a84d74417 before fulfilling a group invite,check if user is already joined/invited (#3436)
Fixes vector-im/riot-web#5645
2019-10-10 13:03:44 +01:00
krombel 2efd050c9d send 404 as http-status when filter-id is unknown to the server (#2380)
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
2019-10-10 12:59:55 +01:00
Erik Johnston 59e0ed8306 Fix py3.5 2019-10-10 12:47:07 +01:00
Erik Johnston c349e3ebaf Fix py3.5 2019-10-10 12:29:38 +01:00
Erik Johnston f735aeec65 sort 2019-10-10 12:20:29 +01:00
Erik Johnston 941edad583 Appease mypy 2019-10-10 12:15:17 +01:00
James 0aee490013 Add snapcraft packaging information (#6084) 2019-10-10 11:59:06 +01:00
Erik Johnston 791a8c559b Add coments 2019-10-10 11:53:57 +01:00
Erik Johnston 9970f955ce Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner 2019-10-10 11:51:50 +01:00
Richard van der Hoff a139420a3c Fix races in room stats (and other) updates. (#6187)
Hopefully this will fix the occasional failures we were seeing in the room directory.

The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.

We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Richard van der Hoff 562b4e51dd Rewrite the user_filter migration again (#6184)
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
Erik Johnston f4571a0497 Newsfile 2019-10-10 11:17:00 +01:00
Erik Johnston 128d5948c4 Fix packaging 2019-10-10 11:16:26 +01:00
Erik Johnston ec0596f2ab Log correct context 2019-10-10 11:11:38 +01:00
Erik Johnston 3e4272961a Test for sentinel commit 2019-10-10 10:58:32 +01:00
Erik Johnston 1d6dd1c294 Move patch_inline_callbacks into synapse/ 2019-10-10 10:53:06 +01:00
Andrew Morgan da815c1f69 Move tag/push rules room upgrade checking ealier (#6155)
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.

The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
2019-10-10 10:06:45 +01:00
Amber Brown f743108a94 Refactor HomeserverConfig so it can be typechecked (#6137) 2019-10-10 09:39:35 +01:00
Hubert Chathi b46cc856ec add changelog 2019-10-09 18:03:40 -04:00
Hubert Chathi 4535a07f4a make version optional in body of e2e backup version update
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
Erik Johnston 5c1f886c75 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner 2019-10-09 16:52:21 +01:00
Erik Johnston c3b34dc32f Newsfile 2019-10-09 16:32:04 +01:00
Erik Johnston 7f18b3d526 Do the update as a background index 2019-10-09 16:03:24 +01:00
Erik Johnston def5413480 Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
Erik Johnston de26678724 Update changelog.d/6185.bugfix
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-09 15:13:02 +01:00
Brendan Abolivier 59d6290ed9 Merge pull request #6178 from matrix-org/babolivier/factor_out_bg_updates
Factor out backgroung updates
2019-10-09 12:29:01 +01:00
Erik Johnston 1d3858371e Disable bytes usage with postgres
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
Erik Johnston e7631d84e6 Fix existing hex encoded json values in DB 2019-10-08 16:18:43 +01:00
Erik Johnston 6b72508d15 Newsfile 2019-10-08 16:18:43 +01:00
Erik Johnston ced4784592 Fix inserting bytes as text 2019-10-08 16:18:43 +01:00
Brendan Abolivier c69324ffb5 Fix RegistrationStore 2019-10-08 14:48:33 +01:00
Brendan Abolivier b1c0a4ceb3 Cleanup client_ips 2019-10-08 14:38:14 +01:00
Brendan Abolivier 8f1b385acc Don't end up with 4 classes in registration 2019-10-08 14:36:33 +01:00
Anshul Angaria 474abf1eb6 add M_TOO_LARGE error code for uploading a too large file (#6151)
Fixes #6109
2019-10-08 13:55:16 +01:00
Andrew Morgan ea7d938bca Remove unused public room list timeout param (#6179)
* Remove unused public room list timeout param

* Add changelog
2019-10-08 13:51:25 +01:00
Richard van der Hoff b94a401852 Fix /federation/v1/state for recent room versions (#6170)
* Fix /federation/v1/state for recent room versions

Turns out this endpoint was completely broken for v3 rooms. Hopefully this
re-signing code is irrelevant nowadays anyway.
2019-10-08 09:35:37 +01:00
Brendan Abolivier 21b5d8b107 Changelog 2019-10-07 18:00:31 +01:00
Brendan Abolivier 66ebea1723 Lint 2019-10-07 17:44:41 +01:00
Richard van der Hoff cc2e19ad4b fix changelog 2019-10-07 17:37:55 +01:00
Brendan Abolivier 0496eafbf4 Move roommember's bg updates to a dedicated store 2019-10-07 17:35:01 +01:00
Brendan Abolivier e106a0e4db Move user_directory's bg updates to a dedicated store 2019-10-07 17:34:45 +01:00
Brendan Abolivier cfccd2d78a Move state's bg updates to a dedicated store 2019-10-07 17:34:39 +01:00
Brendan Abolivier 841054ad96 Move search's bg updates to a dedicated store 2019-10-07 17:34:35 +01:00
Brendan Abolivier 81e6ffb536 Move registration's bg updates to a dedicated store 2019-10-07 17:34:29 +01:00
Brendan Abolivier 54f87e0734 Move media_repository's bg updates to a dedicated store 2019-10-07 17:34:26 +01:00
Brendan Abolivier cef9f6753e Move devices's bg updates to a dedicated store 2019-10-07 17:34:20 +01:00
Brendan Abolivier 2d3b4f42f0 Move deviceinbox's bg updates to a dedicated store 2019-10-07 17:34:16 +01:00
Brendan Abolivier 88957199e7 Move client_ips's bg updates to a dedicated store 2019-10-07 17:34:12 +01:00
Erik Johnston 2cb7466993 Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure
Don't regenerate numeric user ID if registration fails.
2019-10-07 16:56:23 +01:00
Michael Kaye dc795ba709 Log responder we are using. (#6139)
This prevents us logging "Responding to media request with responder %s".
2019-10-07 15:41:25 +01:00
Brendan Abolivier 1992f21a9f Fix changelog for PR #6175 2019-10-07 14:54:36 +01:00
Richard van der Hoff 276ae5c63e add some logging to the rooms stats updates, to try to track down a flaky test (#6167) 2019-10-07 14:41:39 +01:00
Brendan Abolivier 97e2722723 Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
Brendan Abolivier aa7a003074 Changelog 2019-10-07 13:16:54 +01:00
Brendan Abolivier c8e6c308c6 Fix unique_user_filter_index schema update 2019-10-07 13:15:35 +01:00
Erik Johnston 86f4705866 Merge pull request #6159 from matrix-org/erikj/cache_memberships
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
Erik Johnston 5119a4cac7 Fix bug where we didn't pull out event ID 2019-10-07 12:21:17 +01:00
Brendan Abolivier ae0b78cb1f Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
Brendan Abolivier 21d51ab598 Typo 2019-10-04 11:21:24 +01:00
Brendan Abolivier 4676732ca0 Lint 2019-10-04 11:18:28 +01:00
Brendan Abolivier 81d51ce48b Incorporate review 2019-10-04 11:16:19 +01:00
Alexander Maznev 13c4345c84 Update user_filters table to have a unique index, and non-null columns (#1172) 2019-10-04 10:34:16 +01:00
Robert Swain 39b40d6d99 media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
Erik Johnston 6511071837 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships 2019-10-03 17:51:06 +01:00
Richard van der Hoff 66537e10ce add some metrics on the federation sender (#6160) 2019-10-03 17:47:20 +01:00
Erik Johnston 0186ec9df7 Fixup newsfile 2019-10-03 17:46:43 +01:00
Erik Johnston ab8a64772b Remove unused variable 2019-10-03 17:42:32 +01:00
Erik Johnston 4fc60f12de Newsfile 2019-10-03 17:35:50 +01:00
Erik Johnston 693156aaf4 Don't regenerate numeric user ID if registration fails.
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
Erik Johnston 91f61fc6d7 Use the right error.... 2019-10-03 17:28:31 +01:00
Erik Johnston 84691da6c3 pep8 2019-10-03 17:27:18 +01:00
Erik Johnston a9610cdf02 Fixup names and comments 2019-10-03 17:26:56 +01:00
Erik Johnston d89ebf7c25 cachedList descriptor doesn't like typing 2019-10-03 17:23:11 +01:00
Erik Johnston 0ccf0ffc85 Newsfile 2019-10-03 17:12:24 +01:00
Erik Johnston c8145af8a9 Cache room membership lookups in _get_joined_users_from_context 2019-10-03 17:11:04 +01:00
Andrew Morgan 6018bec919 Merge branch 'master' into develop 2019-10-03 13:48:45 +01:00
Andrew Morgan 782dd72037 Merge tag 'v1.4.0'
Synapse 1.4.0 (2019-10-03)
==========================

Bugfixes
--------

- Redact `client_secret` in server logs. ([\#6158](https://github.com/matrix-org/synapse/issues/6158))

Synapse 1.4.0rc2 (2019-10-02)
=============================

Bugfixes
--------

- Fix bug in background update that adds last seen information to the `devices` table, and improve its performance on Postgres. ([\#6135](https://github.com/matrix-org/synapse/issues/6135))
- Fix bad performance of censoring redactions background task. ([\#6141](https://github.com/matrix-org/synapse/issues/6141))
- Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. ([\#6145](https://github.com/matrix-org/synapse/issues/6145))
- Fix exceptions when storing large retry intervals for down remote servers. ([\#6146](https://github.com/matrix-org/synapse/issues/6146))

Internal Changes
----------------

- Fix up sample config entry for `redaction_retention_period` option. ([\#6117](https://github.com/matrix-org/synapse/issues/6117))

Synapse 1.4.0rc1 (2019-09-26)
=============================

Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst#upgrading-to-v140).

Features
--------

- Changes to 3pid verification:
  - Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\#5835](https://github.com/matrix-org/synapse/issues/5835), [\#5940](https://github.com/matrix-org/synapse/issues/5940), [\#5993](https://github.com/matrix-org/synapse/issues/5993), [\#5994](https://github.com/matrix-org/synapse/issues/5994), [\#5868](https://github.com/matrix-org/synapse/issues/5868))
  - Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](https://github.com/matrix-org/matrix-doc/pull/2263). ([\#5876](https://github.com/matrix-org/synapse/issues/5876), [\#5969](https://github.com/matrix-org/synapse/issues/5969), [\#6028](https://github.com/matrix-org/synapse/issues/6028))
  - Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140)). ([\#5897](https://github.com/matrix-org/synapse/issues/5897))
  - Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140). ([\#5964](https://github.com/matrix-org/synapse/issues/5964))
  - Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](https://github.com/matrix-org/matrix-doc/pull/2264). ([\#5974](https://github.com/matrix-org/synapse/issues/5974))
  - Use the v2 Identity Service API for 3PID invites. ([\#5979](https://github.com/matrix-org/synapse/issues/5979))
  - Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\#5980](https://github.com/matrix-org/synapse/issues/5980), [\#6062](https://github.com/matrix-org/synapse/issues/6062))
  - Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\#6011](https://github.com/matrix-org/synapse/issues/6011))
  - Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\#6042](https://github.com/matrix-org/synapse/issues/6042))
  - Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290). ([\#6043](https://github.com/matrix-org/synapse/issues/6043))
  - Add an unstable feature flag for separate add/bind 3pid APIs. ([\#6044](https://github.com/matrix-org/synapse/issues/6044))
  - Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/). ([\#6067](https://github.com/matrix-org/synapse/issues/6067))
  - Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\#6078](https://github.com/matrix-org/synapse/issues/6078))
  - Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\#6079](https://github.com/matrix-org/synapse/issues/6079))
  - Add `m.require_identity_server` flag to /version's unstable_features. ([\#5972](https://github.com/matrix-org/synapse/issues/5972))
- Enhancements to OpenTracing support:
  - Make OpenTracing work in worker mode. ([\#5771](https://github.com/matrix-org/synapse/issues/5771))
  - Pass OpenTracing contexts between servers when transmitting EDUs. ([\#5852](https://github.com/matrix-org/synapse/issues/5852))
  - OpenTracing for device list updates. ([\#5853](https://github.com/matrix-org/synapse/issues/5853))
  - Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\#5856](https://github.com/matrix-org/synapse/issues/5856))
  - Add minimum OpenTracing for client servlets. ([\#5983](https://github.com/matrix-org/synapse/issues/5983))
  - Check at setup that OpenTracing is installed if it's enabled in the config. ([\#5985](https://github.com/matrix-org/synapse/issues/5985))
  - Trace replication send times. ([\#5986](https://github.com/matrix-org/synapse/issues/5986))
  - Include missing OpenTracing contexts in outbout replication requests. ([\#5982](https://github.com/matrix-org/synapse/issues/5982))
  - Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\#5984](https://github.com/matrix-org/synapse/issues/5984))
  - Fix invalid references to None while OpenTracing if the log context slips. ([\#5988](https://github.com/matrix-org/synapse/issues/5988), [\#5991](https://github.com/matrix-org/synapse/issues/5991))
  - OpenTracing for room and e2e keys. ([\#5855](https://github.com/matrix-org/synapse/issues/5855))
  - Add OpenTracing span over HTTP push processing. ([\#6003](https://github.com/matrix-org/synapse/issues/6003))
- Add an admin API to purge old rooms from the database. ([\#5845](https://github.com/matrix-org/synapse/issues/5845))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\#5850](https://github.com/matrix-org/synapse/issues/5850))
- Add support for filtered room-directory search requests over federation ([MSC2197](https://github.com/matrix-org/matrix-doc/pull/2197), in order to allow upcoming room directory query performance improvements. ([\#5859](https://github.com/matrix-org/synapse/issues/5859))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\#5864](https://github.com/matrix-org/synapse/issues/5864))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\#5878](https://github.com/matrix-org/synapse/issues/5878))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\#5884](https://github.com/matrix-org/synapse/issues/5884))
- Add config option to sign remote key query responses with a separate key. ([\#5895](https://github.com/matrix-org/synapse/issues/5895))
- Add support for config templating. ([\#5900](https://github.com/matrix-org/synapse/issues/5900))
- Users with the type of "support" or "bot" are no longer required to consent. ([\#5902](https://github.com/matrix-org/synapse/issues/5902))
- Let synctl accept a directory of config files. ([\#5904](https://github.com/matrix-org/synapse/issues/5904))
- Increase max display name size to 256. ([\#5906](https://github.com/matrix-org/synapse/issues/5906))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\#5914](https://github.com/matrix-org/synapse/issues/5914))
- Redact events in the database that have been redacted for a week. ([\#5934](https://github.com/matrix-org/synapse/issues/5934))
- New prometheus metrics:
  - `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\#5981](https://github.com/matrix-org/synapse/issues/5981))
  - `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\#6005](https://github.com/matrix-org/synapse/issues/6005))
- Give appropriate exit codes when synctl fails. ([\#5992](https://github.com/matrix-org/synapse/issues/5992))
- Apply the federation blacklist to requests to identity servers. ([\#6000](https://github.com/matrix-org/synapse/issues/6000))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\#6012](https://github.com/matrix-org/synapse/issues/6012))
- Add config option to increase ratelimits for room admins redacting messages. ([\#6015](https://github.com/matrix-org/synapse/issues/6015))
- Stop sending federation transactions to servers which have been down for a long time. ([\#6026](https://github.com/matrix-org/synapse/issues/6026))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\#6037](https://github.com/matrix-org/synapse/issues/6037))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\#6073](https://github.com/matrix-org/synapse/issues/6073))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\#6074](https://github.com/matrix-org/synapse/issues/6074))
- Return 403 on `/register/available` if registration has been disabled. ([\#6082](https://github.com/matrix-org/synapse/issues/6082))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\#6090](https://github.com/matrix-org/synapse/issues/6090))
- Add support for pruning old rows in `user_ips` table. ([\#6098](https://github.com/matrix-org/synapse/issues/6098))

Bugfixes
--------

- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\#5633](https://github.com/matrix-org/synapse/issues/5633))
- Fix database index so that different backup versions can have the same sessions. ([\#5857](https://github.com/matrix-org/synapse/issues/5857))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\#5863](https://github.com/matrix-org/synapse/issues/5863))
- Fix stack overflow when recovering an appservice which had an outage. ([\#5885](https://github.com/matrix-org/synapse/issues/5885))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\#5909](https://github.com/matrix-org/synapse/issues/5909))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\#5915](https://github.com/matrix-org/synapse/issues/5915))
- Fix a cache-invalidation bug for worker-based deployments. ([\#5920](https://github.com/matrix-org/synapse/issues/5920))
- Fix admin API for listing media in a room not being available with an external media repo. ([\#5966](https://github.com/matrix-org/synapse/issues/5966))
- Fix list media admin API always returning an error. ([\#5967](https://github.com/matrix-org/synapse/issues/5967))
- Fix room and user stats tracking. ([\#5971](https://github.com/matrix-org/synapse/issues/5971), [\#5998](https://github.com/matrix-org/synapse/issues/5998), [\#6029](https://github.com/matrix-org/synapse/issues/6029))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\#5995](https://github.com/matrix-org/synapse/issues/5995))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\#5996](https://github.com/matrix-org/synapse/issues/5996))
- Only count real users when checking for auto-creation of auto-join room. ([\#6004](https://github.com/matrix-org/synapse/issues/6004))
- Ensure support users can be registered even if MAU limit is reached. ([\#6020](https://github.com/matrix-org/synapse/issues/6020))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\#6024](https://github.com/matrix-org/synapse/issues/6024))
- Fix bug in calculating the federation retry backoff period. ([\#6025](https://github.com/matrix-org/synapse/issues/6025))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\#6053](https://github.com/matrix-org/synapse/issues/6053))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\#6056](https://github.com/matrix-org/synapse/issues/6056))
- Fix logcontext spam on non-Linux platforms. ([\#6059](https://github.com/matrix-org/synapse/issues/6059))
- Ensure query parameters in email validation links are URL-encoded. ([\#6063](https://github.com/matrix-org/synapse/issues/6063))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\#6069](https://github.com/matrix-org/synapse/issues/6069))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\#6092](https://github.com/matrix-org/synapse/issues/6092))
- Add `sid` to `next_link` for email validation. ([\#6097](https://github.com/matrix-org/synapse/issues/6097))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\#6104](https://github.com/matrix-org/synapse/issues/6104))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\#6107](https://github.com/matrix-org/synapse/issues/6107))

Updates to the Docker image
---------------------------

- Avoid changing `UID/GID` if they are already correct. ([\#5970](https://github.com/matrix-org/synapse/issues/5970))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\#6058](https://github.com/matrix-org/synapse/issues/6058))

Improved Documentation
----------------------

- Convert documentation to markdown (from rst) ([\#5849](https://github.com/matrix-org/synapse/issues/5849))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\#5953](https://github.com/matrix-org/synapse/issues/5953))
- Add developer documentation for using SAML2. ([\#6032](https://github.com/matrix-org/synapse/issues/6032))
- Add some notes on rolling back to v1.3.1. ([\#6049](https://github.com/matrix-org/synapse/issues/6049))
- Update the upgrade notes. ([\#6050](https://github.com/matrix-org/synapse/issues/6050))

Deprecations and Removals
-------------------------

- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5877](https://github.com/matrix-org/synapse/issues/5877))
- Deprecate the `trusted_third_party_id_servers` option. ([\#5875](https://github.com/matrix-org/synapse/issues/5875))

Internal Changes
----------------

- Lay the groundwork for structured logging output. ([\#5680](https://github.com/matrix-org/synapse/issues/5680))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\#5844](https://github.com/matrix-org/synapse/issues/5844))
- Remove log line for debugging issue #5407. ([\#5860](https://github.com/matrix-org/synapse/issues/5860))
- Refactor the Appservice scheduler code. ([\#5886](https://github.com/matrix-org/synapse/issues/5886))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\#5892](https://github.com/matrix-org/synapse/issues/5892), [\#6013](https://github.com/matrix-org/synapse/issues/6013))
- Stop populating some unused tables. ([\#5893](https://github.com/matrix-org/synapse/issues/5893), [\#6047](https://github.com/matrix-org/synapse/issues/6047))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\#5894](https://github.com/matrix-org/synapse/issues/5894))
- Improve the logging when we have an error when fetching signing keys. ([\#5896](https://github.com/matrix-org/synapse/issues/5896))
- Add support for database engine-specific schema deltas, based on file extension. ([\#5911](https://github.com/matrix-org/synapse/issues/5911))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\#5922](https://github.com/matrix-org/synapse/issues/5922))
- Add link in sample config to the logging config schema. ([\#5926](https://github.com/matrix-org/synapse/issues/5926))
- Remove unnecessary parentheses in return statements. ([\#5931](https://github.com/matrix-org/synapse/issues/5931))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\#5938](https://github.com/matrix-org/synapse/issues/5938))
- Move Buildkite pipeline config to the pipelines repo. ([\#5943](https://github.com/matrix-org/synapse/issues/5943))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\#5962](https://github.com/matrix-org/synapse/issues/5962))
- Remove left-over methods from v1 registration API. ([\#5963](https://github.com/matrix-org/synapse/issues/5963))
- Cleanup event auth type initialisation. ([\#5975](https://github.com/matrix-org/synapse/issues/5975))
- Clean up dependency checking at setup. ([\#5989](https://github.com/matrix-org/synapse/issues/5989))
- Update OpenTracing docs to use the unified `trace` method. ([\#5776](https://github.com/matrix-org/synapse/issues/5776))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\#6009](https://github.com/matrix-org/synapse/issues/6009))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\#6010](https://github.com/matrix-org/synapse/issues/6010))
- Add a `failure_ts` column to the `destinations` database table. ([\#6016](https://github.com/matrix-org/synapse/issues/6016), [\#6072](https://github.com/matrix-org/synapse/issues/6072))
- Clean up some code in the retry logic. ([\#6017](https://github.com/matrix-org/synapse/issues/6017))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\#6023](https://github.com/matrix-org/synapse/issues/6023))
- Clean up the sample config for SAML authentication. ([\#6064](https://github.com/matrix-org/synapse/issues/6064))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\#6075](https://github.com/matrix-org/synapse/issues/6075))
- Move last-seen info into devices table. ([\#6089](https://github.com/matrix-org/synapse/issues/6089))
- Remove unused parameter to `get_user_id_by_threepid`. ([\#6099](https://github.com/matrix-org/synapse/issues/6099))
- Refactor the user-interactive auth handling. ([\#6105](https://github.com/matrix-org/synapse/issues/6105))
- Refactor code for calculating registration flows. ([\#6106](https://github.com/matrix-org/synapse/issues/6106))
2019-10-03 13:48:21 +01:00
Brendan Abolivier 8a5e8e829b Lint (again) 2019-10-03 11:30:43 +01:00
Brendan Abolivier ebcb6a30d7 Lint 2019-10-03 11:29:07 +01:00
Brendan Abolivier 6527fa18c1 Add test case 2019-10-03 11:24:36 +01:00
Erik Johnston dca7e32d3d Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
Erik Johnston 22a6ffdf91 Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
Erik Johnston 5d8ffdfe61 Merge pull request #6148 from matrix-org/erikj/find_next_generated
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
Erik Johnston 5be4083306 Newsfile 2019-10-02 15:48:36 +01:00
Erik Johnston 7a5f080f91 Fix appservice room list pagination 2019-10-02 15:47:22 +01:00
Erik Johnston 5b9e5c27d8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_room_list_non_federatable 2019-10-02 15:38:54 +01:00
Erik Johnston 824df3ead4 Merge pull request #6152 from matrix-org/erikj/fix_room_list
Fix public room list pagination.
2019-10-02 15:38:05 +01:00
Erik Johnston ed73f04bef Newsfile 2019-10-02 15:24:33 +01:00
Erik Johnston 4c4f44930d Fix not showing non-federatable rooms to remote room list queries 2019-10-02 15:20:36 +01:00
Erik Johnston 8e32240e6b Newsfile 2019-10-02 15:12:17 +01:00
Erik Johnston 03cf4385e0 Fix public room list pagination.
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Brendan Abolivier baf12bc02a Merge branch 'master' into develop 2019-10-02 14:41:02 +01:00
Erik Johnston a5166e4d5f Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Amber Brown 864f144543 Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Brendan Abolivier 24efea338d Changelog 2019-10-02 12:20:03 +01:00
Brendan Abolivier 972c9f65d7 Lint 2019-10-02 12:17:46 +01:00
Andrew Morgan 2a1470cd05 Fix yields and copy instead of move push rules on room upgrade (#6144)
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.

For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.

Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Erik Johnston de1823b521 Newsfile 2019-10-02 11:45:31 +01:00
Erik Johnston d69fd53f74 Bound find_next_generated_user_id DB query.
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
Brendan Abolivier 5705ecaec6 Don't 500 code when trying to exchange a revoked 3PID invite
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).

This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
Erik Johnston ecd254bc49 Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop 2019-10-02 11:08:07 +01:00
Erik Johnston 1d349fb159 Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop 2019-10-01 10:17:24 +01:00
Richard van der Hoff 16cb9a71b8 Drop unused tables (#6115)
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
2019-09-30 09:38:41 +01:00
Andrew Morgan f3451118a6 Edit SimpleHttpClient to reference that header keys can be passed as str or bytes (#6077) 2019-09-27 17:59:18 +01:00
Brendan Abolivier 8f90b0ee48 Merge pull request #6125 from matrix-org/babolivier/deactivation-invite
Reject pending invites on deactivation
2019-09-27 16:43:38 +01:00
Brendan Abolivier 3e42d47a5a Incorporate review 2019-09-27 16:15:01 +01:00
Brendan Abolivier af92110c46 Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
Brendan Abolivier bbe2a0f339 Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
Brendan Abolivier 25a0a36ad9 Update changelog.d/6125.feature
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:24 +01:00
Erik Johnston 6374ca40c2 Update 2019-09-27 15:58:14 +01:00
Brendan Abolivier fbb8ff3088 ok 2019-09-27 15:23:07 +01:00
Brendan Abolivier 873fe7883c Lint 2019-09-27 15:21:03 +01:00
Brendan Abolivier 0804a27c8c Changelog 2019-09-27 15:14:34 +01:00
Brendan Abolivier e94ff67903 Add test to validate the change 2019-09-27 15:14:02 +01:00
Brendan Abolivier 72a2708ac6 Fixup and add some logging 2019-09-27 15:13:39 +01:00
Erik Johnston 132279a46f Patch inlinecallbacks for log contexts 2019-09-27 15:11:14 +01:00
Brendan Abolivier 5257a2fb1c Reject pending invites on deactivation 2019-09-27 14:49:53 +01:00
Andrew Morgan 8c27bc8b60 Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978)
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
Erik Johnston 67ee18daea Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
Richard van der Hoff 54569c787b Kill off half-implemented password-reset via sms (#6101)
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.

See also matrix-org/matrix-doc#2303.
2019-09-26 15:38:25 +01:00
Erik Johnston dc2c97e1a3 isort 2019-09-25 11:59:05 +01:00
Erik Johnston dc01cad690 Add device and appservice tags 2019-09-25 11:59:00 +01:00
Erik Johnston 5c1af6d1b8 Newsfile 2019-09-25 11:42:00 +01:00
Erik Johnston 5d99713854 Add tags for event_id and txn_id in event sending
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
Erik Johnston fde4ce2213 Don't create new span for get_user_by_req
We don't actually care about what happens in `get_user_by_req` and
having it as a separate span means that the entity tag isn't added to
the servlet spans, making it harder to search.
2019-09-25 11:32:41 +01:00
Richard van der Hoff 466866a1d9 Update the issue template for new way of getting server version (#6051)
cf #4878
2019-09-19 15:08:14 +01:00
Pete a136137b2e Update INSTALL.md with void-linux (#5873) 2019-09-19 09:52:59 +01:00
axel simon 1c7df13e7b add explanations on how to actually include an access_token (#6031) 2019-09-13 08:50:17 +01:00
Richard van der Hoff 9eaa5d6d24 README: link to reverse_proxy.rst (#6027) 2019-09-12 21:13:31 +01:00
Hubert Chathi 8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Hubert Chathi 19bb5c8024 Merge pull request #5769 from matrix-org/uhoreg/e2e_cross-signing2-part1
Cross-signing [2/4] - upload/download keys
2019-09-04 16:56:29 -07:00
Hubert Chathi a22d58c96c add user signature stream change cache to slaved device store 2019-09-04 19:32:35 -04:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Hubert Chathi 72d296a7f3 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-08-28 17:33:44 -07:00
Hubert Chathi 96bda56370 black 2019-08-28 17:18:40 -07:00
Hubert Chathi 3b0b22cb05 use stream ID generator instead of timestamp 2019-08-28 17:17:21 -07:00
Hubert Chathi 814f253f1b make isort happy 2019-08-21 13:22:15 -07:00
Hubert Chathi 7c3abc6572 apply PR review suggestions 2019-08-21 13:19:35 -07:00
Hubert Chathi f63ba7a795 Cross-signing [1/4] -- hidden devices (#5759)
* allow devices to be marked as "hidden"

This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
Hubert Chathi 8c9adcc95d fix formatting 2019-08-01 22:09:05 -04:00
Hubert Chathi d28d1e2d1b add changelog 2019-08-01 21:52:35 -04:00
Hubert Chathi fac1cdc562 make changes from PR review 2019-08-01 21:51:19 -04:00
Hubert Chathi 336c546d6a Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-08-01 16:31:40 -04:00
Hubert Chathi d78a4fe156 don't need to return the hidden column any more 2019-08-01 02:16:09 -04:00
Hubert Chathi 73b26f827c really fix queries to work with Postgres
(by going back to not using SQL directly)
2019-07-31 18:37:05 -04:00
Hubert Chathi 430ea08186 PostgreSQL, Y U no like? 2019-07-31 15:38:11 -04:00
Hubert Chathi 185188be03 remove extra SQL query param 2019-07-31 15:18:15 -04:00
Hubert Chathi c1f0a5636c Merge branch 'develop' into cross-signing_hidden 2019-07-30 23:33:06 -04:00
Hubert Chathi bc95890822 Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-07-30 23:21:26 -04:00
Hubert Chathi 2997a91250 add changelog file 2019-07-30 23:14:00 -04:00
Hubert Chathi 781ade836b apply changes from PR review 2019-07-30 23:09:50 -04:00
Hubert Chathi c659b9f94f allow uploading keys for cross-signing 2019-07-25 11:08:24 -04:00
Hubert Chathi d1c7c2a98a allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
215 changed files with 4151 additions and 2476 deletions
-48
View File
@@ -1,48 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2019 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
from tap.parser import Parser
from tap.line import Result, Unknown, Diagnostic
out = ["### TAP Output for " + sys.argv[2]]
p = Parser()
in_error = False
for line in p.parse_file(sys.argv[1]):
if isinstance(line, Result):
if in_error:
out.append("")
out.append("</pre></code></details>")
out.append("")
out.append("----")
out.append("")
in_error = False
if not line.ok and not line.todo:
in_error = True
out.append("FAILURE Test #%d: ``%s``" % (line.number, line.description))
out.append("")
out.append("<details><summary>Show log</summary><code><pre>")
elif isinstance(line, Diagnostic) and in_error:
out.append(line.text)
if out:
for line in out[:-3]:
print(line)
+14 -10
View File
@@ -7,7 +7,7 @@ about: Create a report to help us improve
<!--
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**:
You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)
You will likely get better support more quickly if you ask in ** #synapse:matrix.org ** ;)
This is a bug report template. By following the instructions below and
@@ -44,22 +44,26 @@ those (please be careful to remove any personal or private data). Please surroun
<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->
<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**:
- **Homeserver**:
If not matrix.org:
<!--
What version of Synapse is running?
You can find the Synapse version by inspecting the server headers (replace matrix.org with
your own homeserver domain):
$ curl -v https://matrix.org/_matrix/client/versions 2>&1 | grep "Server:"
-->
- **Version**:
What version of Synapse is running?
- **Install method**:
You can find the Synapse version with this command:
$ curl http://localhost:8008/_synapse/admin/v1/server_version
(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**:
- **Install method**:
<!-- examples: package manager/git clone/pip -->
- **Platform**:
- **Platform**:
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
+2
View File
@@ -7,9 +7,11 @@
*.egg-info
*.lock
*.pyc
*.snap
*.tac
_trial_temp/
_trial_temp*/
/out
# stuff that is likely to exist when you run a server locally
/*.db
+14
View File
@@ -1,3 +1,17 @@
Synapse 1.4.1 (2019-10-18)
==========================
No changes since 1.4.1rc1.
Synapse 1.4.1rc1 (2019-10-17)
=============================
Bugfixes
--------
- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](https://github.com/matrix-org/synapse/commit/5b0e9948eaae801643e594b5abc8ee4b10bd194e))
Synapse 1.4.0 (2019-10-03)
==========================
+7
View File
@@ -349,6 +349,13 @@ sudo pip uninstall py-bcrypt
sudo pip install py-bcrypt
```
### Void Linux
Synapse can be found in the void repositories as 'synapse':
xbps-install -Su
xbps-install -S synapse
### FreeBSD
Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
+1 -3
View File
@@ -47,7 +47,5 @@ prune debian
prune demo/etc
prune docker
prune mypy.ini
prune snap
prune stubs
exclude jenkins*
recursive-exclude jenkins *.sh
+13
View File
@@ -381,3 +381,16 @@ indicate that your server is also issuing far more outgoing federation
requests than can be accounted for by your users' activity, this is a
likely cause. The misbehavior can be worked around by setting
``use_presence: false`` in the Synapse config file.
People can't accept room invitations from me
--------------------------------------------
The typical failure mode here is that you send an invitation to someone
to join a room or direct chat, but when they go to accept it, they get an
error (typically along the lines of "Invalid signature"). They might see
something like the following in their logs::
2019-09-11 19:32:04,271 - synapse.federation.transport.server - 288 - WARNING - GET-11752 - authenticate_request failed: 401: Invalid signature for server <server> with key ed25519:a_EqML: Unable to verify signature for <server>
This is normally caused by a misconfiguration in your reverse-proxy. See
`<docs/reverse_proxy.rst>`_ and double-check that your settings are correct.
+1
View File
@@ -0,0 +1 @@
Update `user_filters` table to have a unique index, and non-null columns. Thanks to @pik for contributing this.
+1
View File
@@ -0,0 +1 @@
Improve quality of thumbnails for 1-bit/8-bit color palette images.
+1
View File
@@ -0,0 +1 @@
Return an HTTP 404 instead of 400 when requesting a filter by ID that is unknown to the server. Thanks to @krombel for contributing this!
+1
View File
@@ -0,0 +1 @@
Fix a problem where users could be invited twice to the same group.
+1
View File
@@ -0,0 +1 @@
Added domain validation when including a list of invitees upon room creation.
+4
View File
@@ -0,0 +1,4 @@
Allow devices to be marked as hidden, for use by features such as cross-signing.
This adds a new field with a default value to the devices field in the database,
and so the database upgrade may take a long time depending on how many devices
are in the database.
+1
View File
@@ -0,0 +1 @@
Allow uploading of cross-signing keys.
+1
View File
@@ -0,0 +1 @@
Move lookup-related functions from RoomMemberHandler to IdentityHandler.
+1
View File
@@ -0,0 +1 @@
Improve performance of the public room list directory.
+1
View File
@@ -0,0 +1 @@
Edit header dicts docstrings in SimpleHttpClient to note that `str` or `bytes` can be passed as header keys.
+1
View File
@@ -0,0 +1 @@
Add snapcraft packaging information. Contributed by @devec0.
+1
View File
@@ -0,0 +1 @@
Kill off half-implemented password-reset via sms.
+1
View File
@@ -0,0 +1 @@
Remove `get_user_by_req` opentracing span and add some tags.
+1
View File
@@ -0,0 +1 @@
Fix bug when uploading a large file: Synapse responds with `M_UNKNOWN` while it should be `M_TOO_LARGE` according to spec. Contributed by Anshul Angaria.
+1
View File
@@ -0,0 +1 @@
CAS login now provides a default display name for users if a `displayname_attribute` is set in the configuration file.
+1
View File
@@ -0,0 +1 @@
Drop some unused database tables.
+1
View File
@@ -0,0 +1 @@
Reject all pending invites for a user during deactivation.
+1
View File
@@ -0,0 +1 @@
Add env var to turn on tracking of log context changes.
+1
View File
@@ -0,0 +1 @@
Refactor configuration loading to allow better typechecking.
+1
View File
@@ -0,0 +1 @@
Log responder when responding to media request.
+1
View File
@@ -0,0 +1 @@
Prevent user push rules being deleted from a room when it is upgraded.
+1
View File
@@ -0,0 +1 @@
Don't 500 when trying to exchange a revoked 3PID invite.
+1
View File
@@ -0,0 +1 @@
Improve performance of `find_next_generated_user_id` DB query.
+1
View File
@@ -0,0 +1 @@
Expand type-checking on modules imported by synapse.config.
+1
View File
@@ -0,0 +1 @@
Improve performance of the public room list directory.
+1
View File
@@ -0,0 +1 @@
Improve performance of the public room list directory.
+1
View File
@@ -0,0 +1 @@
Improve performance of the public room list directory.
+1
View File
@@ -0,0 +1 @@
Fix transferring notifications and tags when joining an upgraded room that is new to your server.
+1
View File
@@ -0,0 +1 @@
Use Postgres ANY for selecting many values.
+1
View File
@@ -0,0 +1 @@
Add more caching to `_get_joined_users_from_context` DB query.
+1
View File
@@ -0,0 +1 @@
Add some metrics on the federation sender.
+1
View File
@@ -0,0 +1 @@
Fix bug where guest account registration can wedge after restart.
+1
View File
@@ -0,0 +1 @@
Add some logging to the rooms stats updates, to try to track down a flaky test.
+1
View File
@@ -0,0 +1 @@
Fix monthly active user reaping where reserved users are specified.
+1
View File
@@ -0,0 +1 @@
Fix /federation/v1/state endpoint for recent room versions.
+1
View File
@@ -0,0 +1 @@
Update `user_filters` table to have a unique index, and non-null columns. Thanks to @pik for contributing this.
+1
View File
@@ -0,0 +1 @@
Make the `synapse_port_db` script create the right indexes on a new PostgreSQL database.
+1
View File
@@ -0,0 +1 @@
Remove unused `timeout` parameter from `_get_public_room_list`.
+1
View File
@@ -0,0 +1 @@
Update `user_filters` table to have a unique index, and non-null columns. Thanks to @pik for contributing this.
+1
View File
@@ -0,0 +1 @@
Fix bug where we were updating censored events as bytes rather than text, occaisonally causing invalid JSON being inserted breaking APIs that attempted to fetch such events.
+1
View File
@@ -0,0 +1 @@
Reject (accidental) attempts to insert bytes into postgres tables.
+1
View File
@@ -0,0 +1 @@
Fix occasional missed updates in the room and user directories.
+1
View File
@@ -0,0 +1 @@
Make `version` optional in body of `PUT /room_keys/version/{version}`, since it's redundant.
+1
View File
@@ -0,0 +1 @@
Add snapcraft packaging information. Contributed by @devec0.
+1
View File
@@ -0,0 +1 @@
Make storage layer responsible for adding device names to key, rather than the handler.
+1
View File
@@ -0,0 +1 @@
Fix tracing of non-JSON APIs, /media, /key etc.
+1
View File
@@ -0,0 +1 @@
Port synapse.rest.admin module to use async/await.
+1
View File
@@ -0,0 +1 @@
Fix logging getting lost for the docker image.
+1
View File
@@ -0,0 +1 @@
Fix bug where presence would not get timed out correctly if a synchrotron worker is used and restarted.
+1
View File
@@ -0,0 +1 @@
Remove some unused event-auth code.
+1
View File
@@ -0,0 +1 @@
synapse_port_db: Add 2 additional BOOLEAN_COLUMNS to be able to convert from database schema v56.
+1
View File
@@ -0,0 +1 @@
Remove Auth.check method.
+1
View File
@@ -0,0 +1 @@
Remove `format_tap.py` script in favour of a perl reimplementation in Sytest's repo.
+11 -24
View File
@@ -1,39 +1,26 @@
# Synapse Docker
FIXME: this is out-of-date as of
https://github.com/matrix-org/synapse/issues/5518. Contributions to bring it up
to date would be welcome.
### Automated configuration
It is recommended that you use Docker Compose to run your containers, including
this image and a Postgres server. A sample ``docker-compose.yml`` is provided,
including example labels for reverse proxying and other artifacts.
Read the section about environment variables and set at least mandatory variables,
then run the server:
```
docker-compose up -d
```
If secrets are not specified in the environment variables, they will be generated
as part of the startup. Please ensure these secrets are kept between launches of the
Docker container, as their loss may require users to log in again.
### Manual configuration
### Configuration
A sample ``docker-compose.yml`` is provided, including example labels for
reverse proxying and other artifacts. The docker-compose file is an example,
please comment/uncomment sections that are not suitable for your usecase.
Specify a ``SYNAPSE_CONFIG_PATH``, preferably to a persistent path,
to use manual configuration. To generate a fresh ``homeserver.yaml``, simply run:
to use manual configuration.
To generate a fresh `homeserver.yaml`, you can use the `generate` command.
(See the [documentation](../../docker/README.md#generating-a-configuration-file)
for more information.) You will need to specify appropriate values for at least the
`SYNAPSE_SERVER_NAME` and `SYNAPSE_REPORT_STATS` environment variables. For example:
```
docker-compose run --rm -e SYNAPSE_SERVER_NAME=my.matrix.host synapse generate
docker-compose run --rm -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes synapse generate
```
(This will also generate necessary signing keys.)
Then, customize your configuration and run the server:
```
+16 -5
View File
@@ -15,13 +15,10 @@ services:
restart: unless-stopped
# See the readme for a full documentation of the environment settings
environment:
- SYNAPSE_SERVER_NAME=my.matrix.host
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_ENABLE_REGISTRATION=yes
- SYNAPSE_LOG_LEVEL=INFO
- POSTGRES_PASSWORD=changeme
- SYNAPSE_CONFIG_PATH=/etc/homeserver.yaml
volumes:
# You may either store all the files in a local folder
- ./matrix-config:/etc
- ./files:/data
# .. or you may split this between different storage points
# - ./files:/data
@@ -35,9 +32,23 @@ services:
- 8448:8448/tcp
# ... or use a reverse proxy, here is an example for traefik:
labels:
# The following lines are valid for Traefik version 1.x:
- traefik.enable=true
- traefik.frontend.rule=Host:my.matrix.Host
- traefik.port=8008
# Alternatively, for Traefik version 2.0:
- traefik.enable=true
- traefik.http.routers.http-synapse.entryPoints=http
- traefik.http.routers.http-synapse.rule=Host(`my.matrix.host`)
- traefik.http.middlewares.https_redirect.redirectscheme.scheme=https
- traefik.http.middlewares.https_redirect.redirectscheme.permanent=true
- traefik.http.routers.http-synapse.middlewares=https_redirect
- traefik.http.routers.https-synapse.entryPoints=https
- traefik.http.routers.https-synapse.rule=Host(`my.matrix.host`)
- traefik.http.routers.https-synapse.service=synapse
- traefik.http.routers.https-synapse.tls=true
- traefik.http.services.synapse.loadbalancer.server.port=8008
- traefik.http.routers.https-synapse.tls.certResolver=le-ssl
db:
image: docker.io/postgres:10-alpine
+6
View File
@@ -1,3 +1,9 @@
matrix-synapse-py3 (1.4.1) stable; urgency=medium
* New synapse release 1.4.1.
-- Synapse Packaging team <packages@matrix.org> Fri, 18 Oct 2019 10:13:27 +0100
matrix-synapse-py3 (1.4.0) stable; urgency=medium
* New synapse release 1.4.0.
+1 -1
View File
@@ -44,7 +44,7 @@ EOF
for DIR in /var/lib/matrix-synapse /var/log/matrix-synapse /etc/matrix-synapse; do
if ! dpkg-statoverride --list --quiet $DIR >/dev/null; then
dpkg-statoverride --force --quiet --update --add $USER nogroup 0755 $DIR
dpkg-statoverride --force-all --quiet --update --add $USER nogroup 0755 $DIR
fi
done
+2
View File
@@ -24,3 +24,5 @@ loggers:
root:
level: {{ SYNAPSE_LOG_LEVEL or "INFO" }}
handlers: [console]
disable_existing_loggers: false
+12
View File
@@ -10,3 +10,15 @@ server admin by updating the database directly, e.g.:
``UPDATE users SET admin = 1 WHERE name = '@foo:bar.com'``
Restarting may be required for the changes to register.
Using an admin access_token
###########################
Many of the API calls listed in the documentation here will require to include an admin `access_token`.
Finding your user's `access_token` is client-dependent, but will usually be shown in the client's settings.
Once you have your `access_token`, to include it in a request, the best option is to add the token to a request header:
``curl --header "Authorization: Bearer <access_token>" <the_rest_of_your_API_request>``
Fore more details, please refer to the complete `matrix spec documentation <https://matrix.org/docs/spec/client_server/r0.5.0#using-access-tokens>`_.
+1
View File
@@ -1220,6 +1220,7 @@ saml2_config:
# enabled: true
# server_url: "https://cas-server.com"
# service_url: "https://homeserver.domain.com:8448"
# #displayname_attribute: name
# #required_attributes:
# # name: value
+12 -4
View File
@@ -4,10 +4,6 @@ plugins=mypy_zope:plugin
follow_imports=skip
mypy_path=stubs
[mypy-synapse.config.homeserver]
# this is a mess because of the metaclass shenanigans
ignore_errors = True
[mypy-zope]
ignore_missing_imports = True
@@ -52,3 +48,15 @@ ignore_missing_imports = True
[mypy-signedjson.*]
ignore_missing_imports = True
[mypy-prometheus_client.*]
ignore_missing_imports = True
[mypy-service_identity.*]
ignore_missing_imports = True
[mypy-daemonize]
ignore_missing_imports = True
[mypy-sentry_sdk]
ignore_missing_imports = True
-58
View File
@@ -1,58 +0,0 @@
from __future__ import print_function
import argparse
import itertools
import json
import sys
from mock import Mock
from synapse.api.auth import Auth
from synapse.events import FrozenEvent
def check_auth(auth, auth_chain, events):
auth_chain.sort(key=lambda e: e.depth)
auth_map = {e.event_id: e for e in auth_chain}
create_events = {}
for e in auth_chain:
if e.type == "m.room.create":
create_events[e.room_id] = e
for e in itertools.chain(auth_chain, events):
auth_events_list = [auth_map[i] for i, _ in e.auth_events]
auth_events = {(e.type, e.state_key): e for e in auth_events_list}
auth_events[("m.room.create", "")] = create_events[e.room_id]
try:
auth.check(e, auth_events=auth_events)
except Exception as ex:
print("Failed:", e.event_id, e.type, e.state_key)
print("Auth_events:", auth_events)
print(ex)
print(json.dumps(e.get_dict(), sort_keys=True, indent=4))
# raise
print("Success:", e.event_id, e.type, e.state_key)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"json", nargs="?", type=argparse.FileType("r"), default=sys.stdin
)
args = parser.parse_args()
js = json.load(args.json)
auth = Auth(Mock())
check_auth(
auth,
[FrozenEvent(d) for d in js["auth_chain"]],
[FrozenEvent(d) for d in js.get("pdus", [])],
)
+2
View File
@@ -55,6 +55,8 @@ BOOLEAN_COLUMNS = {
"local_group_membership": ["is_publicised", "is_admin"],
"e2e_room_keys": ["is_verified"],
"account_validity": ["email_sent"],
"redactions": ["have_censored"],
"room_stats_state": ["is_federatable"],
}
+22
View File
@@ -0,0 +1,22 @@
name: matrix-synapse
base: core18
version: git
summary: Reference Matrix homeserver
description: |
Synapse is the reference Matrix homeserver.
Matrix is a federated and decentralised instant messaging and VoIP system.
grade: stable
confinement: strict
apps:
matrix-synapse:
command: synctl --no-daemonize start $SNAP_COMMON/homeserver.yaml
stop-command: synctl -c $SNAP_COMMON stop
plugs: [network-bind, network]
daemon: simple
parts:
matrix-synapse:
source: .
plugin: python
python-version: python3
+9 -1
View File
@@ -17,6 +17,7 @@
""" This is a reference implementation of a Matrix home server.
"""
import os
import sys
# Check that we're not running on an unsupported Python version.
@@ -35,4 +36,11 @@ try:
except ImportError:
pass
__version__ = "1.4.0"
__version__ = "1.4.1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when
# running the packaging tox test.
from synapse.util.patch_inline_callbacks import do_patch
do_patch()
+4 -19
View File
@@ -84,27 +84,10 @@ class Auth(object):
)
auth_events = yield self.store.get_events(auth_events_ids)
auth_events = {(e.type, e.state_key): e for e in itervalues(auth_events)}
self.check(
event_auth.check(
room_version, event, auth_events=auth_events, do_sig_check=do_sig_check
)
def check(self, room_version, event, auth_events, do_sig_check=True):
""" Checks if this event is correctly authed.
Args:
room_version (str): version of the room
event: the event being checked.
auth_events (dict: event-key -> event): the existing room state.
Returns:
True if the auth checks pass.
"""
with Measure(self.clock, "auth.check"):
event_auth.check(
room_version, event, auth_events, do_sig_check=do_sig_check
)
@defer.inlineCallbacks
def check_joined_room(self, room_id, user_id, current_state=None):
"""Check if the user is currently joined in the room
@@ -179,7 +162,6 @@ class Auth(object):
def get_public_keys(self, invite_event):
return event_auth.get_public_keys(invite_event)
@opentracing.trace
@defer.inlineCallbacks
def get_user_by_req(
self, request, allow_guest=False, rights="access", allow_expired=False
@@ -212,6 +194,7 @@ class Auth(object):
if user_id:
request.authenticated_entity = user_id
opentracing.set_tag("authenticated_entity", user_id)
opentracing.set_tag("appservice_id", app_service.id)
if ip_addr and self.hs.config.track_appservice_user_ips:
yield self.store.insert_client_ip(
@@ -263,6 +246,8 @@ class Auth(object):
request.authenticated_entity = user.to_string()
opentracing.set_tag("authenticated_entity", user.to_string())
if device_id:
opentracing.set_tag("device_id", device_id)
return synapse.types.create_requester(
user, token_id, is_guest, device_id, app_service=app_service
-2
View File
@@ -97,8 +97,6 @@ class EventTypes(object):
class RejectedReason(object):
AUTH_ERROR = "auth_error"
REPLACED = "replaced"
NOT_ANCESTOR = "not_ancestor"
class RoomCreationPreset(object):
+3 -1
View File
@@ -17,6 +17,7 @@
"""Contains exceptions and error codes."""
import logging
from typing import Dict
from six import iteritems
from six.moves import http_client
@@ -61,6 +62,7 @@ class Codes(object):
INCOMPATIBLE_ROOM_VERSION = "M_INCOMPATIBLE_ROOM_VERSION"
WRONG_ROOM_KEYS_VERSION = "M_WRONG_ROOM_KEYS_VERSION"
EXPIRED_ACCOUNT = "ORG_MATRIX_EXPIRED_ACCOUNT"
INVALID_SIGNATURE = "M_INVALID_SIGNATURE"
USER_DEACTIVATED = "M_USER_DEACTIVATED"
@@ -111,7 +113,7 @@ class ProxiedRequestError(SynapseError):
def __init__(self, code, msg, errcode=Codes.UNKNOWN, additional_fields=None):
super(ProxiedRequestError, self).__init__(code, msg, errcode)
if additional_fields is None:
self._additional_fields = {}
self._additional_fields = {} # type: Dict
else:
self._additional_fields = dict(additional_fields)
+4 -1
View File
@@ -12,6 +12,9 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import Dict
import attr
@@ -102,4 +105,4 @@ KNOWN_ROOM_VERSIONS = {
RoomVersions.V4,
RoomVersions.V5,
)
} # type: dict[str, RoomVersion]
} # type: Dict[str, RoomVersion]
+3 -1
View File
@@ -263,7 +263,9 @@ def start(hs, listeners=None):
refresh_certificate(hs)
# Start the tracer
synapse.logging.opentracing.init_tracer(hs.config)
synapse.logging.opentracing.init_tracer( # type: ignore[attr-defined] # noqa
hs.config
)
# It is now safe to start your Synapse.
hs.start_listening(listeners)
+3 -3
View File
@@ -605,13 +605,13 @@ def run(hs):
@defer.inlineCallbacks
def generate_monthly_active_users():
current_mau_count = 0
reserved_count = 0
reserved_users = ()
store = hs.get_datastore()
if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
current_mau_count = yield store.get_monthly_active_count()
reserved_count = yield store.get_registered_reserved_users_count()
reserved_users = yield store.get_registered_reserved_users()
current_mau_gauge.set(float(current_mau_count))
registered_reserved_users_mau_gauge.set(float(reserved_count))
registered_reserved_users_mau_gauge.set(float(len(reserved_users)))
max_mau_gauge.set(float(hs.config.max_mau_value))
def start_generate_monthly_active_users():
+148 -43
View File
@@ -18,7 +18,9 @@
import argparse
import errno
import os
from collections import OrderedDict
from textwrap import dedent
from typing import Any, MutableMapping, Optional
from six import integer_types
@@ -51,7 +53,56 @@ Missing mandatory `server_name` config option.
"""
def path_exists(file_path):
"""Check if a file exists
Unlike os.path.exists, this throws an exception if there is an error
checking if the file exists (for example, if there is a perms error on
the parent dir).
Returns:
bool: True if the file exists; False if not.
"""
try:
os.stat(file_path)
return True
except OSError as e:
if e.errno != errno.ENOENT:
raise e
return False
class Config(object):
"""
A configuration section, containing configuration keys and values.
Attributes:
section (str): The section title of this config object, such as
"tls" or "logger". This is used to refer to it on the root
logger (for example, `config.tls.some_option`). Must be
defined in subclasses.
"""
section = None
def __init__(self, root_config=None):
self.root = root_config
def __getattr__(self, item: str) -> Any:
"""
Try and fetch a configuration option that does not exist on this class.
This is so that existing configs that rely on `self.value`, where value
is actually from a different config section, continue to work.
"""
if item in ["generate_config_section", "read_config"]:
raise AttributeError(item)
if self.root is None:
raise AttributeError(item)
else:
return self.root._get_unclassed_config(self.section, item)
@staticmethod
def parse_size(value):
if isinstance(value, integer_types):
@@ -88,22 +139,7 @@ class Config(object):
@classmethod
def path_exists(cls, file_path):
"""Check if a file exists
Unlike os.path.exists, this throws an exception if there is an error
checking if the file exists (for example, if there is a perms error on
the parent dir).
Returns:
bool: True if the file exists; False if not.
"""
try:
os.stat(file_path)
return True
except OSError as e:
if e.errno != errno.ENOENT:
raise e
return False
return path_exists(file_path)
@classmethod
def check_file(cls, file_path, config_name):
@@ -136,42 +172,106 @@ class Config(object):
with open(file_path) as file_stream:
return file_stream.read()
def invoke_all(self, name, *args, **kargs):
"""Invoke all instance methods with the given name and arguments in the
class's MRO.
class RootConfig(object):
"""
Holder of an application's configuration.
What configuration this object holds is defined by `config_classes`, a list
of Config classes that will be instantiated and given the contents of a
configuration file to read. They can then be accessed on this class by their
section name, defined in the Config or dynamically set to be the name of the
class, lower-cased and with "Config" removed.
"""
config_classes = []
def __init__(self):
self._configs = OrderedDict()
for config_class in self.config_classes:
if config_class.section is None:
raise ValueError("%r requires a section name" % (config_class,))
try:
conf = config_class(self)
except Exception as e:
raise Exception("Failed making %s: %r" % (config_class.section, e))
self._configs[config_class.section] = conf
def __getattr__(self, item: str) -> Any:
"""
Redirect lookups on this object either to config objects, or values on
config objects, so that `config.tls.blah` works, as well as legacy uses
of things like `config.server_name`. It will first look up the config
section name, and then values on those config classes.
"""
if item in self._configs.keys():
return self._configs[item]
return self._get_unclassed_config(None, item)
def _get_unclassed_config(self, asking_section: Optional[str], item: str):
"""
Fetch a config value from one of the instantiated config classes that
has not been fetched directly.
Args:
name (str): Name of function to invoke
asking_section: If this check is coming from a Config child, which
one? This section will not be asked if it has the value.
item: The configuration value key.
Raises:
AttributeError if no config classes have the config key. The body
will contain what sections were checked.
"""
for key, val in self._configs.items():
if key == asking_section:
continue
if item in dir(val):
return getattr(val, item)
raise AttributeError(item, "not found in %s" % (list(self._configs.keys()),))
def invoke_all(self, func_name: str, *args, **kwargs) -> MutableMapping[str, Any]:
"""
Invoke a function on all instantiated config objects this RootConfig is
configured to use.
Args:
func_name: Name of function to invoke
*args
**kwargs
Returns:
list: The list of the return values from each method called
ordered dictionary of config section name and the result of the
function from it.
"""
results = []
for cls in type(self).mro():
if name in cls.__dict__:
results.append(getattr(cls, name)(self, *args, **kargs))
return results
res = OrderedDict()
for name, config in self._configs.items():
if hasattr(config, func_name):
res[name] = getattr(config, func_name)(*args, **kwargs)
return res
@classmethod
def invoke_all_static(cls, name, *args, **kargs):
"""Invoke all static methods with the given name and arguments in the
class's MRO.
def invoke_all_static(cls, func_name: str, *args, **kwargs):
"""
Invoke a static function on config objects this RootConfig is
configured to use.
Args:
name (str): Name of function to invoke
func_name: Name of function to invoke
*args
**kwargs
Returns:
list: The list of the return values from each method called
ordered dictionary of config section name and the result of the
function from it.
"""
results = []
for c in cls.mro():
if name in c.__dict__:
results.append(getattr(c, name)(*args, **kargs))
return results
for config in cls.config_classes:
if hasattr(config, func_name):
getattr(config, func_name)(*args, **kwargs)
def generate_config(
self,
@@ -187,7 +287,8 @@ class Config(object):
tls_private_key_path=None,
acme_domain=None,
):
"""Build a default configuration file
"""
Build a default configuration file
This is used when the user explicitly asks us to generate a config file
(eg with --generate_config).
@@ -242,6 +343,7 @@ class Config(object):
Returns:
str: the yaml config file
"""
return "\n\n".join(
dedent(conf)
for conf in self.invoke_all(
@@ -257,7 +359,7 @@ class Config(object):
tls_certificate_path=tls_certificate_path,
tls_private_key_path=tls_private_key_path,
acme_domain=acme_domain,
)
).values()
)
@classmethod
@@ -444,7 +546,7 @@ class Config(object):
)
(config_path,) = config_files
if not cls.path_exists(config_path):
if not path_exists(config_path):
print("Generating config file %s" % (config_path,))
if config_args.data_directory:
@@ -469,7 +571,7 @@ class Config(object):
open_private_ports=config_args.open_private_ports,
)
if not cls.path_exists(config_dir_path):
if not path_exists(config_dir_path):
os.makedirs(config_dir_path)
with open(config_path, "w") as config_file:
config_file.write("# vim:ft=yaml\n\n")
@@ -518,7 +620,7 @@ class Config(object):
return obj
def parse_config_dict(self, config_dict, config_dir_path, data_dir_path):
def parse_config_dict(self, config_dict, config_dir_path=None, data_dir_path=None):
"""Read the information from the config dict into this Config object.
Args:
@@ -607,3 +709,6 @@ def find_config_files(search_paths):
else:
config_files.append(config_path)
return config_files
__all__ = ["Config", "RootConfig"]
+135
View File
@@ -0,0 +1,135 @@
from typing import Any, List, Optional
from synapse.config import (
api,
appservice,
captcha,
cas,
consent_config,
database,
emailconfig,
groups,
jwt_config,
key,
logger,
metrics,
password,
password_auth_providers,
push,
ratelimiting,
registration,
repository,
room_directory,
saml2_config,
server,
server_notices_config,
spam_checker,
stats,
third_party_event_rules,
tls,
tracer,
user_directory,
voip,
workers,
)
class ConfigError(Exception): ...
MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS: str
MISSING_REPORT_STATS_SPIEL: str
MISSING_SERVER_NAME: str
def path_exists(file_path: str): ...
class RootConfig:
server: server.ServerConfig
tls: tls.TlsConfig
database: database.DatabaseConfig
logging: logger.LoggingConfig
ratelimit: ratelimiting.RatelimitConfig
media: repository.ContentRepositoryConfig
captcha: captcha.CaptchaConfig
voip: voip.VoipConfig
registration: registration.RegistrationConfig
metrics: metrics.MetricsConfig
api: api.ApiConfig
appservice: appservice.AppServiceConfig
key: key.KeyConfig
saml2: saml2_config.SAML2Config
cas: cas.CasConfig
jwt: jwt_config.JWTConfig
password: password.PasswordConfig
email: emailconfig.EmailConfig
worker: workers.WorkerConfig
authproviders: password_auth_providers.PasswordAuthProviderConfig
push: push.PushConfig
spamchecker: spam_checker.SpamCheckerConfig
groups: groups.GroupsConfig
userdirectory: user_directory.UserDirectoryConfig
consent: consent_config.ConsentConfig
stats: stats.StatsConfig
servernotices: server_notices_config.ServerNoticesConfig
roomdirectory: room_directory.RoomDirectoryConfig
thirdpartyrules: third_party_event_rules.ThirdPartyRulesConfig
tracer: tracer.TracerConfig
config_classes: List = ...
def __init__(self) -> None: ...
def invoke_all(self, func_name: str, *args: Any, **kwargs: Any): ...
@classmethod
def invoke_all_static(cls, func_name: str, *args: Any, **kwargs: Any) -> None: ...
def __getattr__(self, item: str): ...
def parse_config_dict(
self,
config_dict: Any,
config_dir_path: Optional[Any] = ...,
data_dir_path: Optional[Any] = ...,
) -> None: ...
read_config: Any = ...
def generate_config(
self,
config_dir_path: str,
data_dir_path: str,
server_name: str,
generate_secrets: bool = ...,
report_stats: Optional[str] = ...,
open_private_ports: bool = ...,
listeners: Optional[Any] = ...,
database_conf: Optional[Any] = ...,
tls_certificate_path: Optional[str] = ...,
tls_private_key_path: Optional[str] = ...,
acme_domain: Optional[str] = ...,
): ...
@classmethod
def load_or_generate_config(cls, description: Any, argv: Any): ...
@classmethod
def load_config(cls, description: Any, argv: Any): ...
@classmethod
def add_arguments_to_parser(cls, config_parser: Any) -> None: ...
@classmethod
def load_config_with_parser(cls, parser: Any, argv: Any): ...
def generate_missing_files(
self, config_dict: dict, config_dir_path: str
) -> None: ...
class Config:
root: RootConfig
def __init__(self, root_config: Optional[RootConfig] = ...) -> None: ...
def __getattr__(self, item: str, from_root: bool = ...): ...
@staticmethod
def parse_size(value: Any): ...
@staticmethod
def parse_duration(value: Any): ...
@staticmethod
def abspath(file_path: Optional[str]): ...
@classmethod
def path_exists(cls, file_path: str): ...
@classmethod
def check_file(cls, file_path: str, config_name: str): ...
@classmethod
def ensure_directory(cls, dir_path: str): ...
@classmethod
def read_file(cls, file_path: str, config_name: str): ...
def read_config_files(config_files: List[str]): ...
def find_config_files(search_paths: List[str]): ...
+2
View File
@@ -18,6 +18,8 @@ from ._base import Config
class ApiConfig(Config):
section = "api"
def read_config(self, config, **kwargs):
self.room_invite_state_types = config.get(
"room_invite_state_types",
+5 -2
View File
@@ -13,6 +13,7 @@
# limitations under the License.
import logging
from typing import Dict
from six import string_types
from six.moves.urllib import parse as urlparse
@@ -29,6 +30,8 @@ logger = logging.getLogger(__name__)
class AppServiceConfig(Config):
section = "appservice"
def read_config(self, config, **kwargs):
self.app_service_config_files = config.get("app_service_config_files", [])
self.notify_appservices = config.get("notify_appservices", True)
@@ -56,8 +59,8 @@ def load_appservices(hostname, config_files):
return []
# Dicts of value -> filename
seen_as_tokens = {}
seen_ids = {}
seen_as_tokens = {} # type: Dict[str, str]
seen_ids = {} # type: Dict[str, str]
appservices = []
+2
View File
@@ -16,6 +16,8 @@ from ._base import Config
class CaptchaConfig(Config):
section = "captcha"
def read_config(self, config, **kwargs):
self.recaptcha_private_key = config.get("recaptcha_private_key")
self.recaptcha_public_key = config.get("recaptcha_public_key")
+5
View File
@@ -22,17 +22,21 @@ class CasConfig(Config):
cas_server_url: URL of CAS server
"""
section = "cas"
def read_config(self, config, **kwargs):
cas_config = config.get("cas_config", None)
if cas_config:
self.cas_enabled = cas_config.get("enabled", True)
self.cas_server_url = cas_config["server_url"]
self.cas_service_url = cas_config["service_url"]
self.cas_displayname_attribute = cas_config.get("displayname_attribute")
self.cas_required_attributes = cas_config.get("required_attributes", {})
else:
self.cas_enabled = False
self.cas_server_url = None
self.cas_service_url = None
self.cas_displayname_attribute = None
self.cas_required_attributes = {}
def generate_config_section(self, config_dir_path, server_name, **kwargs):
@@ -43,6 +47,7 @@ class CasConfig(Config):
# enabled: true
# server_url: "https://cas-server.com"
# service_url: "https://homeserver.domain.com:8448"
# #displayname_attribute: name
# #required_attributes:
# # name: value
"""
+5 -2
View File
@@ -73,8 +73,11 @@ DEFAULT_CONFIG = """\
class ConsentConfig(Config):
def __init__(self):
super(ConsentConfig, self).__init__()
section = "consent"
def __init__(self, *args):
super(ConsentConfig, self).__init__(*args)
self.user_consent_version = None
self.user_consent_template_dir = None
+2
View File
@@ -21,6 +21,8 @@ from ._base import Config
class DatabaseConfig(Config):
section = "database"
def read_config(self, config, **kwargs):
self.event_cache_size = self.parse_size(config.get("event_cache_size", "10K"))
+2
View File
@@ -28,6 +28,8 @@ from ._base import Config, ConfigError
class EmailConfig(Config):
section = "email"
def read_config(self, config, **kwargs):
# TODO: We should separate better the email configuration from the notification
# and account validity config.
+2
View File
@@ -17,6 +17,8 @@ from ._base import Config
class GroupsConfig(Config):
section = "groups"
def read_config(self, config, **kwargs):
self.enable_group_creation = config.get("enable_group_creation", False)
self.group_creation_prefix = config.get("group_creation_prefix", "")
+35 -33
View File
@@ -14,6 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from ._base import RootConfig
from .api import ApiConfig
from .appservice import AppServiceConfig
from .captcha import CaptchaConfig
@@ -46,36 +47,37 @@ from .voip import VoipConfig
from .workers import WorkerConfig
class HomeServerConfig(
ServerConfig,
TlsConfig,
DatabaseConfig,
LoggingConfig,
RatelimitConfig,
ContentRepositoryConfig,
CaptchaConfig,
VoipConfig,
RegistrationConfig,
MetricsConfig,
ApiConfig,
AppServiceConfig,
KeyConfig,
SAML2Config,
CasConfig,
JWTConfig,
PasswordConfig,
EmailConfig,
WorkerConfig,
PasswordAuthProviderConfig,
PushConfig,
SpamCheckerConfig,
GroupsConfig,
UserDirectoryConfig,
ConsentConfig,
StatsConfig,
ServerNoticesConfig,
RoomDirectoryConfig,
ThirdPartyRulesConfig,
TracerConfig,
):
pass
class HomeServerConfig(RootConfig):
config_classes = [
ServerConfig,
TlsConfig,
DatabaseConfig,
LoggingConfig,
RatelimitConfig,
ContentRepositoryConfig,
CaptchaConfig,
VoipConfig,
RegistrationConfig,
MetricsConfig,
ApiConfig,
AppServiceConfig,
KeyConfig,
SAML2Config,
CasConfig,
JWTConfig,
PasswordConfig,
EmailConfig,
WorkerConfig,
PasswordAuthProviderConfig,
PushConfig,
SpamCheckerConfig,
GroupsConfig,
UserDirectoryConfig,
ConsentConfig,
StatsConfig,
ServerNoticesConfig,
RoomDirectoryConfig,
ThirdPartyRulesConfig,
TracerConfig,
]
+2
View File
@@ -23,6 +23,8 @@ MISSING_JWT = """Missing jwt library. This is required for jwt login.
class JWTConfig(Config):
section = "jwt"
def read_config(self, config, **kwargs):
jwt_config = config.get("jwt_config", None)
if jwt_config:
+2
View File
@@ -92,6 +92,8 @@ class TrustedKeyServer(object):
class KeyConfig(Config):
section = "key"
def read_config(self, config, config_dir_path, **kwargs):
# the signing key can be specified inline or in a separate file
if "signing_key" in config:
+4 -3
View File
@@ -68,9 +68,6 @@ handlers:
filters: [context]
loggers:
synapse:
level: INFO
synapse.storage.SQL:
# beware: increasing this to DEBUG will make synapse log sensitive
# information such as access tokens.
@@ -79,11 +76,15 @@ loggers:
root:
level: INFO
handlers: [file, console]
disable_existing_loggers: false
"""
)
class LoggingConfig(Config):
section = "logging"
def read_config(self, config, **kwargs):
self.log_config = self.abspath(config.get("log_config"))
self.no_redirect_stdio = config.get("no_redirect_stdio", False)
+2
View File
@@ -34,6 +34,8 @@ class MetricsFlags(object):
class MetricsConfig(Config):
section = "metrics"
def read_config(self, config, **kwargs):
self.enable_metrics = config.get("enable_metrics", False)
self.report_stats = config.get("report_stats", None)
+2
View File
@@ -20,6 +20,8 @@ class PasswordConfig(Config):
"""Password login configuration
"""
section = "password"
def read_config(self, config, **kwargs):
password_config = config.get("password_config", {})
if password_config is None:
+5 -1
View File
@@ -13,6 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import Any, List
from synapse.util.module_loader import load_module
from ._base import Config
@@ -21,8 +23,10 @@ LDAP_PROVIDER = "ldap_auth_provider.LdapAuthProvider"
class PasswordAuthProviderConfig(Config):
section = "authproviders"
def read_config(self, config, **kwargs):
self.password_providers = []
self.password_providers = [] # type: List[Any]
providers = []
# We want to be backwards compatible with the old `ldap_config`
+2
View File
@@ -18,6 +18,8 @@ from ._base import Config
class PushConfig(Config):
section = "push"
def read_config(self, config, **kwargs):
push_config = config.get("push", {})
self.push_include_content = push_config.get("include_content", True)
+2
View File
@@ -36,6 +36,8 @@ class FederationRateLimitConfig(object):
class RatelimitConfig(Config):
section = "ratelimiting"
def read_config(self, config, **kwargs):
# Load the new-style messages config if it exists. Otherwise fall back
+4
View File
@@ -24,6 +24,8 @@ from synapse.util.stringutils import random_string_with_symbols
class AccountValidityConfig(Config):
section = "accountvalidity"
def __init__(self, config, synapse_config):
self.enabled = config.get("enabled", False)
self.renew_by_email_enabled = "renew_at" in config
@@ -77,6 +79,8 @@ class AccountValidityConfig(Config):
class RegistrationConfig(Config):
section = "registration"
def read_config(self, config, **kwargs):
self.enable_registration = bool(
strtobool(str(config.get("enable_registration", False)))
+5 -2
View File
@@ -15,6 +15,7 @@
import os
from collections import namedtuple
from typing import Dict, List
from synapse.python_dependencies import DependencyException, check_requirements
from synapse.util.module_loader import load_module
@@ -61,7 +62,7 @@ def parse_thumbnail_requirements(thumbnail_sizes):
Dictionary mapping from media type string to list of
ThumbnailRequirement tuples.
"""
requirements = {}
requirements = {} # type: Dict[str, List]
for size in thumbnail_sizes:
width = size["width"]
height = size["height"]
@@ -77,6 +78,8 @@ def parse_thumbnail_requirements(thumbnail_sizes):
class ContentRepositoryConfig(Config):
section = "media"
def read_config(self, config, **kwargs):
# Only enable the media repo if either the media repo is enabled or the
@@ -130,7 +133,7 @@ class ContentRepositoryConfig(Config):
#
# We don't create the storage providers here as not all workers need
# them to be started.
self.media_storage_providers = []
self.media_storage_providers = [] # type: List[tuple]
for provider_config in storage_providers:
# We special case the module "file_system" so as not to need to
+2
View File
@@ -19,6 +19,8 @@ from ._base import Config, ConfigError
class RoomDirectoryConfig(Config):
section = "roomdirectory"
def read_config(self, config, **kwargs):
self.enable_room_list_search = config.get("enable_room_list_search", True)

Some files were not shown because too many files have changed in this diff Show More