1
0

Compare commits

...

170 Commits

Author SHA1 Message Date
Erik Johnston
6b2d6fdd33 Add some debugging 2020-05-15 15:33:11 +01:00
Erik Johnston
d263a4de02 Enable moving event persistence off of master 2020-05-14 17:25:42 +01:00
Erik Johnston
66c1dff3ba Use new writers config 2020-05-14 17:25:41 +01:00
Erik Johnston
96b6023e3b Make location of events writer configurable 2020-05-14 17:25:09 +01:00
Erik Johnston
452019064c Allow ReplicationRestResource to be added to workers 2020-05-14 17:25:09 +01:00
Erik Johnston
7c8e09bcf1 Add a worker store for search insertion 2020-05-14 17:25:09 +01:00
Erik Johnston
e7f5ac4ed8 Fix lint 2020-05-14 17:09:58 +01:00
Erik Johnston
208ab7b135 Fix typing and add assertion. 2020-05-14 17:09:58 +01:00
Erik Johnston
41f558ccf7 Newsfile 2020-05-14 17:09:58 +01:00
Erik Johnston
342796d6ac Move push rules ID gen to push rules worker 2020-05-14 17:09:58 +01:00
Erik Johnston
bc3fc3927f Move events ID gens to EventWorkerStore 2020-05-14 17:09:58 +01:00
Erik Johnston
d67a8b5455 Move repliction event stream handling out of slave store 2020-05-14 17:09:58 +01:00
Erik Johnston
4734a7bbe4 Move EventStream handling into default ReplicationDataHandler (#7493)
This is so that the logic can happen on both master and workers when we move event persistence out.
2020-05-14 14:01:39 +01:00
Erik Johnston
1de36407d1 Add instance_map config and route replication calls (#7495) 2020-05-14 14:00:58 +01:00
Richard van der Hoff
dede23ff1e Merge tag 'v1.13.0rc2' into develop
Synapse 1.13.0rc2 (2020-05-14)
==============================

Bugfixes
--------

- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))

Internal Changes
----------------

- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
2020-05-14 11:46:38 +01:00
Richard van der Hoff
75fbc1a0c6 1.13.0rc2 2020-05-14 11:10:11 +01:00
Richard van der Hoff
623abbeb8d 1.13.0rc2 2020-05-14 11:01:24 +01:00
Richard van der Hoff
a0e063387d Stop get_joined_users corruption from custom statuses (#7376)
Fix a bug where the `get_joined_users` cache could be corrupted by custom
status events (or other state events with a state_key matching the user ID).

The bug was introduced by #2229, but has largely gone unnoticed since then.

Fixes #7099, #7373.
2020-05-14 10:07:54 +01:00
Patrick Cloke
5d64fefd6c Do not validate that the client dict is stable during UI Auth. (#7483)
This backs out some of the validation for the client dictionary and logs if
this changes during a user interactive authentication session instead.
2020-05-13 14:26:44 -04:00
Erik Johnston
1124111a12 Allow censoring of events to happen on workers. (#7492)
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
2020-05-13 17:15:40 +01:00
Paul Tötterman
46cb2550bb Fix copypasted comment (#7477)
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2020-05-13 16:55:43 +01:00
Erik Johnston
18c1e52d82 Clean up replication unit tests. (#7490) 2020-05-13 16:01:47 +01:00
Erik Johnston
00ba9c48bf Spelling 2020-05-13 13:38:51 +01:00
Erik Johnston
782e4e64df Shuffle persist event data store functions. (#7440)
The aim here is to get to a stage where we have a `PersistEventStore` that holds all the write methods used during event persistence, so that we can take that class out of the `DataStore` mixin and instansiate it separately. This will allow us to instansiate it on processes other than master, while also ensuring it is only available on processes that are configured to write to events stream.

This is a bit of an architectural change, where we end up with multiple classes per data store (rather than one per data store we have now). We end up having:

1. Storage classes that provide high level APIs that can talk to multiple data stores.
2. Data store modules that consist of classes that must point at the same database instance.
3. Classes in a data store that can be instantiated on processes depending on config.
2020-05-13 13:38:22 +01:00
Patrick Cloke
edd3b0747c Fix new flake8 errors (#7489)
This is a cherry-pick of 1a1da60ad2 (#7470)
to the release-v1.13.0 branch.
2020-05-13 08:24:50 -04:00
Erik Johnston
7ee24c5674 Have all instances correctly respond to REPLICATE command. (#7475)
Before all streams were only written to from master, so only master needed to respond to `REPLICATE` commands.

Before all instances wrote to the cache invalidation stream, but didn't respond to `REPLICATE`. This was a bug, which could lead to missed rows from cache invalidation stream if an instance is restarted, however all the caches would be empty in that case so it wasn't a problem.
2020-05-13 10:27:02 +01:00
Erik Johnston
8ca79613e6 Fix Redis reconnection logic (#7482)
Proactively send out `POSITION` commands (as if we had just received a `REPLICATE`) when we connect to Redis. This is important as other instances won't notice we've connected to issue a `REPLICATE` command (unlike for direct TCP connections). This is only currently an issue if master process reconnects without restarting (if it restarts then it won't have written anything and so other instances probably won't have missed anything).
2020-05-13 09:57:15 +01:00
Patrick Cloke
51fb0fc2e5 Update documentation about SSO mapping providers (#7458) 2020-05-12 10:51:07 -04:00
Erik Johnston
1a1da60ad2 Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Patrick Cloke
8c8858e124 Convert federation handler to async/await. (#7459) 2020-05-11 15:12:46 -04:00
Patrick Cloke
be309d99cf Convert search code to async/await. (#7460) 2020-05-11 15:12:39 -04:00
Amber Brown
7cb8b4bc67 Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Andrew Morgan
a8580c5f19 Remove unused store method get_hosts_in_room (#7448) 2020-05-11 16:55:57 +01:00
Andrew Morgan
5cf758cdd6 Merge branch 'release-v1.13.0' into develop
* release-v1.13.0:
  Don't UPGRADE database rows
  RST indenting
  Put rollback instructions in upgrade notes
  Fix changelog typo
  Oh yeah, RST
  Absolute URL it is then
  Fix upgrade notes link
  Provide summary of upgrade issues in changelog. Fix )
  Move next version notes from changelog to upgrade notes
  Changelog fixes
  1.13.0rc1
  Documentation on setting up redis (#7446)
  Rework UI Auth session validation for registration (#7455)
  Fix errors from malformed log line (#7454)
  Drop support for redis.dbid (#7450)
2020-05-11 16:46:33 +01:00
Andrew Morgan
fa4af2c3af Don't UPGRADE database rows 2020-05-11 16:08:37 +01:00
Andrew Morgan
e0caeedab3 RST indenting 2020-05-11 16:05:23 +01:00
Andrew Morgan
8cced49764 Put rollback instructions in upgrade notes 2020-05-11 16:04:00 +01:00
Andrew Morgan
647a995b96 Fix changelog typo 2020-05-11 15:40:36 +01:00
Andrew Morgan
c3416c888a Oh yeah, RST 2020-05-11 15:37:51 +01:00
Andrew Morgan
b41d7b3969 Absolute URL it is then 2020-05-11 15:36:32 +01:00
Andrew Morgan
3916c655f8 Fix upgrade notes link 2020-05-11 15:34:59 +01:00
Andrew Morgan
1f73d28b6c Provide summary of upgrade issues in changelog. Fix ) 2020-05-11 15:33:53 +01:00
Andrew Morgan
71c9307509 Move next version notes from changelog to upgrade notes 2020-05-11 15:10:08 +01:00
Andrew Morgan
200ad02624 Changelog fixes 2020-05-11 14:56:29 +01:00
Andrew Morgan
20ffaa7209 1.13.0rc1 2020-05-11 14:54:38 +01:00
Neil Johnson
85155654c5 Documentation on setting up redis (#7446) 2020-05-11 13:21:15 +01:00
Patrick Cloke
0ad6d28b0d Rework UI Auth session validation for registration (#7455)
Be less strict about validation of UI authentication sessions during
registration to match client expecations.
2020-05-08 16:08:58 -04:00
Andrew Morgan
67feea8044 Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Quentin Gliech
616af44137 Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Manuel Stahl
a4a5ec4096 Add room details admin endpoint (#7317) 2020-05-07 15:33:07 -04:00
Richard van der Hoff
aa5aa6f96a Fix errors from malformed log line (#7454) 2020-05-07 19:51:38 +01:00
Richard van der Hoff
da9b2db3af Drop support for redis.dbid (#7450)
Since we only use pubsub, the dbid is irrelevant.
2020-05-07 16:46:15 +01:00
Brendan Abolivier
5bb26b7c4f Merge branch 'release-v1.13.0' into develop 2020-05-07 17:31:19 +02:00
Patrick Cloke
9e0384dd3f Fixes typo (bellow -> below) (#7449) 2020-05-07 09:31:06 -04:00
Patrick Cloke
22246919e3 Add more type hints to SAML handler. (#7445) 2020-05-07 09:30:45 -04:00
Erik Johnston
d7983b63a6 Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Brendan Abolivier
2929ce29d6 Merge pull request #7398 from Starbix/alpine-3.11
Update docker runtime image to Alpine v3.11
2020-05-07 11:56:56 +02:00
Brendan Abolivier
d9b8d27494 Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Patrick Cloke
d7c2df2fa3 Improve per-block CPU and DB usage metrics (#7426) 2020-05-06 16:43:39 -04:00
Andrew Morgan
4162c39dcf Port group attestation renewal slow down from matrix-org-hotfixes (#7442) 2020-05-06 20:21:38 +01:00
Richard van der Hoff
e053c86a96 Make redis go faster with hiredis (#7439)
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
2020-05-06 17:36:46 +01:00
Richard van der Hoff
62ee862119 Merge branch 'release-v1.13.0' into develop 2020-05-06 15:56:03 +01:00
Andrew Morgan
aee9130a83 Stop Auth methods from polling the config on every req. (#7420) 2020-05-06 15:54:58 +01:00
Richard van der Hoff
fa0b2bd28d Merge pull request #7428 from matrix-org/rav/cross_signing_keys_cache
Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
2020-05-06 12:00:01 +01:00
Richard van der Hoff
16b67c404d Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
... mostly because the latter has a cache.
2020-05-06 11:59:19 +01:00
Richard van der Hoff
db5f9031b7 Fix batching for fetching cross-signing keys
There's no point carefully dividing a list into batches, and then completely
ignoring the batches.
2020-05-06 11:59:19 +01:00
Richard van der Hoff
2e0c46ca07 Merge branch 'release-v1.13.0' into develop 2020-05-06 11:58:31 +01:00
Richard van der Hoff
79007a42b2 Merge pull request #7429 from matrix-org/rav/upsert_for_device_list
use an upsert to update device_lists_outbound_last_success
2020-05-06 11:53:18 +01:00
Richard van der Hoff
30a19daa02 Merge branch 'develop' into rav/upsert_for_device_list 2020-05-06 11:43:11 +01:00
Richard van der Hoff
e48361545d use an upsert to update device_lists_outbound_last_success 2020-05-06 11:41:23 +01:00
Richard van der Hoff
0f6ebf393d Better type annotations for simple_upsert_txn
most of these params don't really need to be lists.
2020-05-06 11:41:23 +01:00
Erik Johnston
b26f3e582c Merge pull request #7423 from matrix-org/erikj/faster_device_lists_fetch
Speed up fetching device lists changes in sync.
2020-05-06 11:14:13 +01:00
Richard van der Hoff
c255b0ffdc Merge pull request #7427 from matrix-org/rav/fix_dropped_messages
Fix lost events on replication reconnection
2020-05-06 10:54:25 +01:00
Richard van der Hoff
a8c17da245 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 23:01:12 +01:00
Richard van der Hoff
1242267316 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 22:38:44 +01:00
Richard van der Hoff
7bf788ac73 changelog 2020-05-05 22:38:16 +01:00
Richard van der Hoff
7f7eedbebb Wait for a POSITION on the right connection before accepting RDATA
... otherwise we can believe we're up to date when we're not.
2020-05-05 22:38:16 +01:00
Brendan Abolivier
5b8023dc7f Move logs about discarded RDATA to debug (#7421) 2020-05-05 21:07:33 +02:00
Richard van der Hoff
d78265af0c Wait to subscribe before sending REPLICATE 2020-05-05 19:31:37 +01:00
Richard van der Hoff
13dd458b8d Merge branch 'release-v1.13.0' into erikj/faster_device_lists_fetch 2020-05-05 18:14:00 +01:00
Richard van der Hoff
714560e325 Update changelog.d/7423.misc 2020-05-05 18:03:59 +01:00
Erik Johnston
79fe3e068b Newsfile 2020-05-05 17:40:29 +01:00
Erik Johnston
f9073893af Speed up fetching device lists changes in sync.
Currently we copy `users_who_share_room` needlessly about three times,
which is expensive when the set is large (which it can easily be).
2020-05-05 17:40:29 +01:00
Richard van der Hoff
16b1a34e80 Fix typing annotations in synapse/federation (#7382)
We're pretty close to having mypy working for `synapse.federation`, so let's
finish the job.
2020-05-05 14:27:13 +01:00
Patrick Cloke
fe69fb6263 Add backwards compatibility codepath to LoggingContext. (#7408) 2020-05-05 09:21:34 -04:00
Erik Johnston
7941a70fa8 Fix bug in EventContext.deserialize. (#7393)
This caused `prev_state_ids` to be incorrect if the state event was not
replacing an existing state entry.
2020-05-05 14:17:27 +01:00
Richard van der Hoff
d5aa7d93ed Fix catchup-on-reconnect for the Federation Stream (#7374)
looks like we managed to break this during the refactorathon.
2020-05-05 14:15:57 +01:00
Erik Johnston
8123b2f909 Add MultiWriterIdGenerator. (#7281)
This will be used to coordinate stream IDs across multiple writers.

Functions as the equivalent of both `StreamIdGenerator` and
`SlavedIdTracker`.
2020-05-04 17:17:45 +01:00
Brendan Abolivier
15aa09bbe6 Merge branch 'release-v1.13.0' into develop 2020-05-04 16:33:56 +02:00
Brendan Abolivier
9858d5c362 Fix ordering in MANIFEST.in 2020-05-04 16:33:30 +02:00
Brendan Abolivier
ad088716bc Merge pull request #7404 from matrix-org/babolivier/fix_manifest
Fix MANIFEST.in
2020-05-04 16:24:15 +02:00
Brendan Abolivier
068da604c2 Fix MANIFEST.in
An update of check-manifest shone some light on some issues with MANIFEST.in, specifically that we didn't ignore/prune the contrib directory, and that we were using prune instead of exclude for files. This fixes both issues.

Fixes #7403
2020-05-04 15:18:06 +02:00
Erik Johnston
350421e058 Fix redis password support. (#7401)
We forgot to set the password on the subscriber connection, as well as
not calling super methods for overridden connectionMade/connectionLost
functions.
2020-05-04 14:04:09 +01:00
Patrick Cloke
eab59d758d Convert the room handler to async/await. (#7396) 2020-05-04 07:43:52 -04:00
Cédric Laubacher
a251e0f4ba Update runtime docker image to Alpine v3.11 2020-05-03 16:07:24 +02:00
Patrick Cloke
032e5a2aca Convert synapse.server_notices to async/await. (#7394) 2020-05-01 15:28:59 -04:00
Patrick Cloke
b0cbc57375 Convert the synapse.notifier module to async/await. (#7395) 2020-05-01 15:14:49 -04:00
Richard van der Hoff
97ef1471c6 Wait for current_state_events_membership before populate_stats_process_rooms (#7387)
populate_stats_process_rooms was added in #5971 / v1.4.0; current_state_events_membership was added in #5706 / v1.3.0.

Fixes #7380.
2020-05-01 18:05:58 +01:00
Erik Johnston
0e719f2398 Thread through instance name to replication client. (#7369)
For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
2020-05-01 17:19:56 +01:00
Erik Johnston
3085cde577 Use stream.current_token() and remove stream_positions() (#7172)
We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
2020-05-01 15:21:35 +01:00
Andrew Morgan
6b22921b19 async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Andrew Morgan
2e8955f4a6 Further improvements to requesting the public rooms list on a homeserver which has it set to private (#7368) 2020-05-01 15:15:08 +01:00
Richard van der Hoff
b2dba06079 Workaround for assertion errors from db_query_to_update_function (#7378)
Hopefully this is no worse than what we have on master...
2020-05-01 09:25:16 +01:00
Patrick Cloke
627b0f5f27 Persist user interactive authentication sessions (#7302)
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
2020-04-30 13:47:49 -04:00
Andrew Morgan
9d8ecc9e6c Apply federation check for /publicRooms with filter list (#7367) 2020-04-30 11:38:07 +01:00
Erik Johnston
37f6823f5b Add instance name to RDATA/POSITION commands (#7364)
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
2020-04-29 16:23:08 +01:00
Erik Johnston
3eab76ad43 Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352)
For direct TCP connections we need the master to relay REMOTE_SERVER_UP
commands to the other connections so that all instances get notified
about it. The old implementation just relayed to all connections,
assuming that sending back to the original sender of the command was
safe. This is not true for redis, where commands sent get echoed back to
the sender, which was causing master to effectively infinite loop
sending and then re-receiving REMOTE_SERVER_UP commands that it sent.

The fix is to ensure that we only relay to *other* connections and not
to the connection we received the notification from.

Fixes #7334.
2020-04-29 14:10:59 +01:00
Richard van der Hoff
c2e1a2110f Fix limit logic for EventsStream (#7358)
* Factor out functions for injecting events into database

I want to add some more flexibility to the tools for injecting events into the
database, and I don't want to clutter up HomeserverTestCase with them, so let's
factor them out to a new file.

* Rework TestReplicationDataHandler

This wasn't very easy to work with: the mock wrapping was largely superfluous,
and it's useful to be able to inspect the received rows, and clear out the
received list.

* Fix AssertionErrors being thrown by EventsStream

Part of the problem was that there was an off-by-one error in the assertion,
but also the limit logic was too simple. Fix it all up and add some tests.
2020-04-29 12:30:36 +01:00
Andrew Morgan
eeef9633af Fix fallback value for account_threepid_delegates.email (#7316) 2020-04-29 11:30:06 +01:00
Andrew Morgan
c58ae367d8 Clean up admin api docs (#7361) 2020-04-28 20:06:03 +01:00
Manuel Stahl
04dd7d182d Return total number of users and profile attributes in admin users endpoint (#6881)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-04-28 18:19:36 +01:00
Erik Johnston
fce663889b Add some replication tests (#7278)
Specifically some tests for the typing stream, which means we test streams that fetch missing updates via HTTP (rather than via the DB).

We also shuffle things around a bit so that we create two separate `HomeServer` objects, rather than trying to insert a slaved store into places.

Note: `test_typing.py` is heavily inspired by `test_receipts.py`
2020-04-28 17:42:03 +01:00
Andrew Morgan
ce207aa0ff Fix typo 'datbases' in ConfigError 2020-04-28 16:53:10 +01:00
Richard van der Hoff
fb8ff79efd Fix collation for postgres for unit tests (#7359)
When running the UTs against a postgres deatbase, we need to set the collation
correctly.
2020-04-28 14:21:48 +01:00
Erik Johnston
38919b521e Run replication streamers on workers (#7146)
Currently we never write to streams from workers, but that will change soon
2020-04-28 13:34:12 +01:00
Richard van der Hoff
07337fe30b Fix incorrect metrics reporting for renew_attestations (#7344)
We need to wait for the renewals to finish, so that the metrics are correctly
reported.
2020-04-27 22:20:10 +01:00
Brendan Abolivier
036fab5d8a Document monitoring workers (#7357)
It doesn't seem to be documented anywhere and means that you suddenly start losing metrics without any obvious reason when you go from monolith to workers (e.g. #7312).
2020-04-27 21:36:47 +02:00
lub
aa2492907f Add some explanation to application_services.md (#7091)
Signed-off-by: Simon Körner <git@lubiland.de>
2020-04-27 15:03:09 +01:00
Brendan Abolivier
cc9eceb00d Don't crash when one of the configuration files is empty (#7341)
If the admin adds a `.yaml` file that's either empty or doesn't parse into a dict to a config directory (e.g. `conf.d` for debs installs), stuff like https://github.com/matrix-org/synapse/issues/7322 would happen. This PR checks that the file is correctly parsed into a dict, or ignores it with a warning if it parses into any other type (including `None` for empty files).

Fixes https://github.com/matrix-org/synapse/issues/7322
2020-04-27 15:01:03 +02:00
Patrick Cloke
7bfe0902ce Add documentation to the sample config about the templates for SSO. (#7343) 2020-04-24 15:03:49 -04:00
Patrick Cloke
33bceb7f70 Convert some of the federation handler methods to async/await. (#7338) 2020-04-24 14:36:38 -04:00
Richard van der Hoff
69a1ac00b2 Merge pull request #7337 from matrix-org/rav/fix_update_limit_assertion
Fix assertions being thrown by the EventsStream update function
2020-04-24 14:00:29 +01:00
Richard van der Hoff
3655eafe85 changelog 2020-04-24 13:59:21 +01:00
Richard van der Hoff
ce428a1abe Fix EventsStream raising assertions when it falls behind
Figuring out how to correctly limit updates from this stream without dropping
entries is far more complicated than just counting the number of rows being
returned. We need to consider each query separately and, if any one query hits
the limit, truncate the results from the others.

I think this also fixes some potentially long-standing bugs where events or
state changes could get missed if we hit the limit on either query.
2020-04-24 13:59:21 +01:00
Patrick Cloke
68384d96fd Merge branch 'master' into develop 2020-04-23 12:04:50 -04:00
Patrick Cloke
204664d1ad Merge tag 'v1.12.4'
Synapse v1.12.4

Features:

* Always send users their own device updates. (#7160)
* Add support for handling GET requests for account_data on a worker. (#7311)

Bugfixes:

* Fix a bug that prevented cross-signing with users on worker-mode synapses. (#7255)
* Do not treat display names as globs in push rules. (#7271)
* Fix a bug with cross-signing devices belonging to remote users who did not share a
  room with any user on the local homeserver. (#7289)
2020-04-23 12:03:33 -04:00
Patrick Cloke
ce9b62e13f 1.12.4 2020-04-23 10:59:10 -04:00
Richard van der Hoff
9cbdfb3a2f Make it clear that the limit for an update_function is a target 2020-04-23 15:45:12 +01:00
Richard van der Hoff
23b28266ac Remove 'limit' param from get_repl_stream_updates API
there doesn't seem to be much point in passing this limit all around, since
both sides agree it's meant to be 100.
2020-04-23 15:44:35 +01:00
Brendan Abolivier
2e3b9a0fcb Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token""
This reverts commit 1adf6a5587.
2020-04-23 11:23:53 +02:00
Brendan Abolivier
fb825759e3 Merge branch 'master' into develop 2020-04-23 11:23:33 +02:00
Brendan Abolivier
1adf6a5587 Revert "Merge pull request #7315 from matrix-org/babolivier/request_token"
This reverts commit 6f4319368b, reversing
changes made to 0d775fcc2d.
2020-04-23 11:23:10 +02:00
Brendan Abolivier
6f4319368b Merge pull request #7315 from matrix-org/babolivier/request_token
Config option to inhibit 3PID errors on /requestToken
2020-04-23 10:38:57 +02:00
Richard van der Hoff
71a1abb8a1 Stop the master relaying USER_SYNC for other workers (#7318)
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.

In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.

Fixes (I hope) #7257.
2020-04-22 22:39:04 +01:00
Brendan Abolivier
69ad7cc13b Config option to inhibit 3PID errors on /requestToken
Adds a request_token_inhibit_errors configuration flag (disabled by
default) which, if enabled, change the behaviour of all /requestToken
endpoints so that they return a 200 and a fake sid if the 3PID was/was
not found associated with an account (depending on the endpoint),
instead of an error.

Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-04-22 23:38:42 +02:00
Erik Johnston
841c581c40 Fix replication metrics when using redis (#7325) 2020-04-22 16:26:19 +01:00
Richard van der Hoff
f16beaa969 Merge tag 'v1.12.4rc1' into develop
Synapse 1.12.4rc1 (2020-04-22)
==============================

Features
--------

- Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
- Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))

Bugfixes
--------

- Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
- Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
- Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
2020-04-22 14:54:08 +01:00
Richard van der Hoff
ba0aac5e44 formatting for the changelog 2020-04-22 14:50:51 +01:00
Richard van der Hoff
82d8b1dd1f Another go at fixing one-word commands (#7326)
I messed this up last time I tried (#7239 / e13c6c7).
2020-04-22 14:34:31 +01:00
Richard van der Hoff
dc8003f921 1.12.4rc1 2020-04-22 14:29:39 +01:00
Richard van der Hoff
83af1079d6 1.12.4rc1 2020-04-22 14:28:23 +01:00
Richard van der Hoff
7c7618c7e3 fix changelog name 2020-04-22 14:01:11 +01:00
Richard van der Hoff
13683a3a22 Extend StreamChangeCache to support multiple entities per stream ID (#7303)
First some background: StreamChangeCache is used to keep track of what "entities" have 
changed since a given stream ID. So for example, we might use it to keep track of when the last
to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2].

Now, it turns out that StreamChangeCache didn't support more than one thing being changed at
a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send
to-device messages to more than one user at a time.

As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped
ok with having multiple things changing on the same stream ID, and it seems we never actually
use the methods which don't work on the stream change caches where we allow multiple
changes at the same stream ID. But that feels horribly fragile, hence: let's update
StreamChangeCache to properly support this, and add some typing and some more tests while
we're at it.

[1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301
[2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
2020-04-22 13:45:40 +01:00
Dirk Klimpel
6b6685db9f Extend room admin api with additional attributes (#7225) 2020-04-22 13:38:41 +01:00
Richard van der Hoff
2aa5bf13c8 Merge branch 'release-v1.12.4' into develop 2020-04-22 13:09:23 +01:00
Erik Johnston
51f7eaf908 Add ability to run replication protocol over redis. (#7040)
This is configured via the `redis` config options.
2020-04-22 13:07:41 +01:00
Patrick Cloke
51f358e2fe Do not treat display names as globs for push rules. (#7271) 2020-04-22 13:07:12 +01:00
Michael Kaye
5308239d5d Reduce logging verbosity of URL cache cleanup. (#7295) 2020-04-22 07:45:16 -04:00
Andrew Morgan
f89ad3b6df Query missing cross-signing keys on local sig upload (#7289) 2020-04-22 12:29:36 +01:00
Andrew Morgan
ff5604e7f1 import urllib.parse when using urllib.parse.quote (#7319) 2020-04-22 12:18:51 +01:00
Michael Kaye
336989a57f Reduce federation logging on success (#7321)
Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
2020-04-22 11:18:18 +01:00
Richard van der Hoff
556566f0b8 Fix changelog file
I updated the PR and forgot to update the changelog.
2020-04-21 13:20:16 +01:00
Richard van der Hoff
974c0d726a Support GET account_data requests on a worker (#7311) 2020-04-21 10:46:30 +01:00
Richard van der Hoff
461f01ad43 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:55:39 +01:00
Richard van der Hoff
d41c8f6d4d Revert "Query missing cross-signing keys on local sig upload"
This was incorrectly merged to the release branch before it was ready.

This reverts commit 72fe2affb6.
2020-04-20 17:54:35 +01:00
Richard van der Hoff
a46ff43319 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:42:27 +01:00
David Baker
40f79f58bf Always send the user updates to their own device list (#7160) 2020-04-20 17:20:38 +01:00
Lars Franke
13917232d5 Fix indention in generated config file (#7300)
Also adjust sample_config.yaml

Signed-off-by: Lars Franke <frcl@mailbox.org>
2020-04-20 16:51:27 +01:00
Patrick Cloke
f5ea8b48bd Reject unknown UI auth sessions (instead of silently generating a new one) (#7268) 2020-04-20 08:54:42 -04:00
Richard van der Hoff
0f8f02bc39 On catchup, process each row with its own stream id (#7286)
Other parts of the code (such as the StreamChangeCache) assume that there will
not be multiple changes with the same stream id.

This code was introduced in #7024, and I hope this fixes #7206.
2020-04-20 11:43:29 +01:00
Patrick Cloke
054c231e58 Use a template for the SSO success page to allow for customization. (#7279) 2020-04-17 13:34:55 -04:00
James
701788a227 Added explicit Python build tools to snap requirements (#7213)
Signed-off-by: James Hebden <james@ec0.io>
2020-04-17 17:28:00 +01:00
Andrew Morgan
72fe2affb6 Query missing cross-signing keys on local sig upload
Add changelog

Save retrieved keys to the db

lint

Fix and de-brittle remote result dict processing

Use query_user_devices instead, assume only master, self_signing key types

Make changelog more useful

Remove very specific exception handling

Wrap get_verify_key_from_cross_signing_key in a try/except

Note that _get_e2e_cross_signing_verify_key can raise a SynapseError

lint

Add comment explaining why this is useful

Only fetch master and self_signing key types

Fix log statements, docstrings

Remove extraneous items from remote query try/except

lint

Factor key retrieval out into a separate function

Send device updates, modeled after SigningKeyEduUpdater._handle_signing_key_updates

Update method docstring
2020-04-17 15:47:49 +01:00
Richard van der Hoff
67ff7b8ba0 Improve type checking in replication.tcp.Stream (#7291)
The general idea here is to get rid of the type: ignore annotations on all of the current_token and update_function assignments, which would have caught #7290.

After a bit of experimentation, it seems like the least-awful way to do this is to pass the offending functions in as parameters to the Stream constructor. Unfortunately that means that the concrete implementations no longer have the same constructor signature as Stream itself, which means that it gets hard to correctly annotate STREAMS_MAP.

I've also introduced a couple of new types, to take out some duplication.
2020-04-17 14:49:55 +01:00
nataraj-hates-MS-for-stealing-github
0d775fcc2d Improve example TURN configuration in documentation (#7284) 2020-04-17 08:04:23 -04:00
Tristan Lins
c07fca9e2f Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Patrick Cloke
01294e6b3a Do not treat display names as globs for push rules. (#7271) 2020-04-16 10:52:55 -04:00
Richard van der Hoff
d7d42387f5 Fix 'generator object is not subscriptable' error (#7290)
Some of the query functions return generators rather than lists, so we can't
index into the result. Happily we already have a copy of the results.

(think this was introduced in #7024)
2020-04-16 14:37:06 +01:00
Richard van der Hoff
6fb63d6426 Add some warnings and notes about slow upgrade (#7275)
Fixes #7144.
2020-04-15 12:40:43 +01:00
Andrew Morgan
ac6a84818f Only register devices edu handler on the master process (#7255) 2020-04-14 11:36:24 +01:00
334 changed files with 12208 additions and 4889 deletions

View File

@@ -1,14 +1,196 @@
Next version
============
Synapse 1.13.0rc2 (2020-05-14)
==============================
* Two new templates (`sso_auth_confirm.html` and `sso_account_deactivated.html`)
were added to Synapse. If your Synapse is configured to use SSO and a custom
`sso_redirect_confirm_template_dir` configuration then these templates will
need to be duplicated into that directory.
Bugfixes
--------
* Plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi`
should update to using the async/await version `complete_sso_login_async` which
includes additional checks. The non-async version is considered deprecated.
- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
Internal Changes
----------------
- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
Synapse 1.13.0rc1 (2020-05-11)
==============================
This release brings some potential changes necessary for certain
configurations of Synapse:
* If your Synapse is configured to use SSO and have a custom
`sso_redirect_confirm_template_dir` configuration option set, you will need
to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and
`sso_account_deactivated.html` templates into that directory.
* Synapse plugins using the `complete_sso_login` method of
`synapse.module_api.ModuleApi` should instead switch to the async/await
version, `complete_sso_login_async`, which includes additional checks. The
former version is now deprecated.
* A bug was introduced in Synapse 1.4.0 which could cause the room directory
to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or
earlier, to versions between v1.4.0 and v1.12.x.
Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes
and for general upgrade guidance.
Features
--------
- Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006))
- Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009))
- Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454))
- Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051))
- Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096))
- Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343))
- Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118))
- Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235))
- Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153))
- Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191))
- Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230))
- Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265))
- Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315))
- Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422))
Bugfixes
--------
- Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573))
- Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634))
- Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639))
- Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368))
- Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946))
- Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455))
- Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089))
- Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109))
- Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117))
- Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133))
- Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150))
- Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151))
- Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155))
- Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159))
- Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177))
- Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178))
- Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199))
- Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203))
- Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259))
- Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249))
- Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260))
- Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268))
- Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274))
- Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302))
- Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316))
- Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358))
- Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341))
- Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344))
- Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367))
- Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387))
- Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393))
Improved Documentation
----------------------
- Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892))
- Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988))
- Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091))
- Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107))
- Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119))
- Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141))
- Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147))
- Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167))
- Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171))
- Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234))
- Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248))
- Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251))
- Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272))
- Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357))
- Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361))
Deprecations and Removals
-------------------------
- Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137))
Internal Changes
----------------
- Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446))
- Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881))
- Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010))
- Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011))
- Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421))
- Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241))
- De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115))
- Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116))
- Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426))
- Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128))
- Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136))
- Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146))
- Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157))
- Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158))
- Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172))
- Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185))
- Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187))
- Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188))
- Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190))
- Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192))
- Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193))
- Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207))
- Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213))
- Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219))
- Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225))
- Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228))
- Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233))
- Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236))
- Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237))
- Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243))
- Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261))
- Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278))
- Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291))
- Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295))
- Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300))
- Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303))
- Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319))
- Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321))
- Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338))
- Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359))
- Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363))
- Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364))
- Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369))
- Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394))
- Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395))
- Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404))
- Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420))
- Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423))
- Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442))
Synapse 1.12.4 (2020-04-23)
===========================
No significant changes.
Synapse 1.12.4rc1 (2020-04-22)
==============================
Features
--------
- Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
- Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))
Bugfixes
--------
- Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
- Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
- Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
Synapse 1.12.3 (2020-04-03)
===========================
@@ -42,12 +224,19 @@ Bugfixes
Synapse 1.12.0 (2020-03-23)
===========================
No significant changes since 1.12.0rc1.
Debian packages and Docker images are rebuilt using the latest versions of
dependency libraries, including Twisted 20.3.0. **Please see security advisory
below**.
Potential slow database update during upgrade
---------------------------------------------
Synapse 1.12.0 includes a database update which is run as part of the upgrade,
and which may take some time (several hours in the case of a large
server). Synapse will not respond to HTTP requests while this update is taking
place. For imformation on seeing if you are affected, and workaround if you
are, see the [upgrade notes](UPGRADE.rst#upgrading-to-v1120).
Security advisory
-----------------

View File

@@ -30,23 +30,24 @@ recursive-include synapse/static *.gif
recursive-include synapse/static *.html
recursive-include synapse/static *.js
exclude Dockerfile
exclude .codecov.yml
exclude .coveragerc
exclude .dockerignore
exclude test_postgresql.sh
exclude .editorconfig
exclude Dockerfile
exclude mypy.ini
exclude sytest-blacklist
exclude test_postgresql.sh
include pyproject.toml
recursive-include changelog.d *
prune .buildkite
prune .circleci
prune .codecov.yml
prune .coveragerc
prune .github
prune contrib
prune debian
prune demo/etc
prune docker
prune mypy.ini
prune snap
prune stubs

View File

@@ -75,6 +75,139 @@ for example:
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to v1.13.0
====================
Incorrect database migration in old synapse versions
----------------------------------------------------
A bug was introduced in Synapse 1.4.0 which could cause the room directory to
be incomplete or empty if Synapse was upgraded directly from v1.2.1 or
earlier, to versions between v1.4.0 and v1.12.x.
This will *not* be a problem for Synapse installations which were:
* created at v1.4.0 or later,
* upgraded via v1.3.x, or
* upgraded straight from v1.2.1 or earlier to v1.13.0 or later.
If completeness of the room directory is a concern, installations which are
affected can be repaired as follows:
1. Run the following sql from a `psql` or `sqlite3` console:
.. code:: sql
INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
('populate_stats_process_rooms', '{}', 'current_state_events_membership');
INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
2. Restart synapse.
New Single Sign-on HTML Templates
---------------------------------
New templates (``sso_auth_confirm.html``, ``sso_auth_success.html``, and
``sso_account_deactivated.html``) were added to Synapse. If your Synapse is
configured to use SSO and a custom ``sso_redirect_confirm_template_dir``
configuration then these templates will need to be copied from
`synapse/res/templates <synapse/res/templates>`_ into that directory.
Synapse SSO Plugins Method Deprecation
--------------------------------------
Plugins using the ``complete_sso_login`` method of
``synapse.module_api.ModuleApi`` should update to using the async/await
version ``complete_sso_login_async`` which includes additional checks. The
non-async version is considered deprecated.
Rolling back to v1.12.4 after a failed upgrade
----------------------------------------------
v1.13.0 includes a lot of large changes. If something problematic occurs, you
may want to roll-back to a previous version of Synapse. Because v1.13.0 also
includes a new database schema version, reverting that version is also required
alongside the generic rollback instructions mentioned above. In short, to roll
back to v1.12.4 you need to:
1. Stop the server
2. Decrease the schema version in the database:
.. code:: sql
UPDATE schema_version SET version = 57;
3. Downgrade Synapse by following the instructions for your installation method
in the "Rolling back to older versions" section above.
Upgrading to v1.12.0
====================
This version includes a database update which is run as part of the upgrade,
and which may take some time (several hours in the case of a large
server). Synapse will not respond to HTTP requests while this update is taking
place.
This is only likely to be a problem in the case of a server which is
participating in many rooms.
0. As with all upgrades, it is recommended that you have a recent backup of
your database which can be used for recovery in the event of any problems.
1. As an initial check to see if you will be affected, you can try running the
following query from the `psql` or `sqlite3` console. It is safe to run it
while Synapse is still running.
.. code:: sql
SELECT MAX(q.v) FROM (
SELECT (
SELECT ej.json AS v
FROM state_events se INNER JOIN event_json ej USING (event_id)
WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
LIMIT 1
) FROM rooms WHERE rooms.room_version IS NULL
) q;
This query will take about the same amount of time as the upgrade process: ie,
if it takes 5 minutes, then it is likely that Synapse will be unresponsive for
5 minutes during the upgrade.
If you consider an outage of this duration to be acceptable, no further
action is necessary and you can simply start Synapse 1.12.0.
If you would prefer to reduce the downtime, continue with the steps below.
2. The easiest workaround for this issue is to manually
create a new index before upgrading. On PostgreSQL, his can be done as follows:
.. code:: sql
CREATE INDEX CONCURRENTLY tmp_upgrade_1_12_0_index
ON state_events(room_id) WHERE type = 'm.room.create';
The above query may take some time, but is also safe to run while Synapse is
running.
We assume that no SQLite users have databases large enough to be
affected. If you *are* affected, you can run a similar query, omitting the
``CONCURRENTLY`` keyword. Note however that this operation may in itself cause
Synapse to stop running for some time. Synapse admins are reminded that
`SQLite is not recommended for use outside a test
environment <https://github.com/matrix-org/synapse/blob/master/README.rst#using-postgresql>`_.
3. Once the index has been created, the ``SELECT`` query in step 1 above should
complete quickly. It is therefore safe to upgrade to Synapse 1.12.0.
4. Once Synapse 1.12.0 has successfully started and is responding to HTTP
requests, the temporary index can be removed:
.. code:: sql
DROP INDEX tmp_upgrade_1_12_0_index;
Upgrading to v1.10.0
====================

1
changelog.d/6391.feature Normal file
View File

@@ -0,0 +1 @@
Synapse's cache factor can now be configured in `homeserver.yaml` by the `caches.global_factor` setting. Additionally, `caches.per_cache_factors` controls the cache factors for individual caches.

View File

@@ -1 +0,0 @@
Add benchmarks for LruCache.

View File

@@ -1 +0,0 @@
Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak.

View File

@@ -1 +0,0 @@
Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm.

View File

@@ -1 +0,0 @@
Fix missing field `default` when fetching user-defined push rules.

View File

@@ -1 +0,0 @@
Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`.

View File

@@ -1 +0,0 @@
Improve error responses when accessing remote public room lists.

View File

@@ -1 +0,0 @@
Transfer alias mappings on room upgrade.

View File

@@ -1 +0,0 @@
Improve the documentation for database configuration.

View File

@@ -1 +0,0 @@
Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata.

View File

@@ -1 +0,0 @@
Set `Referrer-Policy` header to `no-referrer` on media downloads.

View File

@@ -1 +0,0 @@
Change device list streams to have one row per ID.

View File

@@ -1 +0,0 @@
Remove concept of a non-limited stream.

View File

@@ -1 +0,0 @@
Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users.

View File

@@ -1 +0,0 @@
Ensure that a user inteactive authentication session is tied to a single request.

View File

@@ -1 +0,0 @@
Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors.

View File

@@ -1 +0,0 @@
Add options to prevent users from changing their profile or associated 3PIDs.

View File

@@ -1 +0,0 @@
Support SSO in the user interactive authentication workflow.

View File

@@ -1 +0,0 @@
Update pre-built package name for FreeBSD.

View File

@@ -1 +0,0 @@
Return the proper error (M_BAD_ALIAS) when a non-existant canonical alias is provided.

View File

@@ -1 +0,0 @@
Convert some of synapse.rest.media to async/await.

View File

@@ -1 +0,0 @@
De-duplicate / remove unused REST code for login and auth.

View File

@@ -1 +0,0 @@
Convert `*StreamRow` classes to inner classes.

View File

@@ -1 +0,0 @@
Fix a bug which meant that groups updates were not correctly replicated between workers.

View File

@@ -1 +0,0 @@
Allow server admins to define and enforce a password policy (MSC2000).

View File

@@ -1 +0,0 @@
Update postgres docs with login troubleshooting information.

View File

@@ -1 +0,0 @@
Clean up some LoggingContext code.

View File

@@ -1 +0,0 @@
Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage.

View File

@@ -1 +0,0 @@
Fix starting workers when federation sending not split out.

View File

@@ -1 +0,0 @@
Refactored the CAS authentication logic to a separate class.

View File

@@ -1 +0,0 @@
Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`.

View File

@@ -1 +0,0 @@
Clean up INSTALL.md a bit.

View File

@@ -1 +0,0 @@
Add documentation for running a local CAS server for testing.

View File

@@ -1 +0,0 @@
Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param.

View File

@@ -1 +0,0 @@
Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response.

View File

@@ -1 +0,0 @@
Improve the support for SSO authentication on the login fallback page.

View File

@@ -1 +0,0 @@
Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set.

View File

@@ -1 +0,0 @@
Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo.

View File

@@ -1 +0,0 @@
Add tests for outbound device pokes.

View File

@@ -1 +0,0 @@
Fix device list update stream ids going backward.

View File

@@ -1 +0,0 @@
Fix excessive CPU usage by `prune_old_outbound_device_pokes` job.

View File

@@ -1 +0,0 @@
Always send users their own device updates.

View File

@@ -1 +0,0 @@
Improve README.md by being explicit about public IP recommendation for TURN relaying.

View File

@@ -1 +0,0 @@
Fix a small typo in the `metrics_flags` config option.

View File

@@ -1 +0,0 @@
Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature.

View File

@@ -1 +0,0 @@
Fix a bug which could cause incorrect 'cyclic dependency' error.

View File

@@ -1 +0,0 @@
Clean up some LoggingContext code.

View File

@@ -1 +0,0 @@
Clean up some LoggingContext code.

View File

@@ -1 +0,0 @@
Convert some of synapse.rest.media to async/await.

View File

@@ -1 +0,0 @@
Move client command handling out of TCP protocol.

View File

@@ -1 +0,0 @@
Support SSO in the user interactive authentication workflow.

View File

@@ -1 +0,0 @@
Move server command handling out of TCP protocol.

View File

@@ -1 +0,0 @@
Fix consistency of HTTP status codes reported in log lines.

View File

@@ -1 +0,0 @@
Only run one background database update at a time.

View File

@@ -1 +0,0 @@
Admin users are no longer required to be in a room to create an alias for it.

View File

@@ -1 +0,0 @@
Remove sent outbound device list pokes from the database.

View File

@@ -1 +0,0 @@
Add a background database update job to clear out duplicate `device_lists_outbound_pokes`.

View File

@@ -1 +0,0 @@
Move catchup of replication streams logic to worker.

View File

@@ -1 +0,0 @@
Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent.

View File

@@ -1 +0,0 @@
Fix some worker-mode replication handling not being correctly recorded in CPU usage stats.

View File

@@ -1 +0,0 @@
Remove some extraneous debugging log lines.

View File

@@ -1 +0,0 @@
Add typing information to federation server code.

View File

@@ -1 +0,0 @@
Move catchup of replication streams logic to worker.

View File

@@ -1 +0,0 @@
Unblacklist '/upgrade creates a new room' sytest for workers.

View File

@@ -1 +0,0 @@
Require admin privileges to enable room encryption by default. This does not affect existing rooms.

View File

@@ -1 +0,0 @@
Remove redundant checks on `daemonize` from synctl.

View File

@@ -1 +0,0 @@
Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution.

View File

@@ -1 +0,0 @@
Improve the support for SSO authentication on the login fallback page.

View File

@@ -1 +0,0 @@
Upgrade jQuery to v3.4.1 on fallback login/registration pages.

View File

@@ -1 +0,0 @@
Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible.

View File

@@ -1 +0,0 @@
Add documentation to the `password_providers` config option. Add known password provider implementations to docs.

View File

@@ -1 +0,0 @@
Do not allow a deactivated user to login via SSO.

View File

@@ -1 +0,0 @@
Convert some of synapse.rest.media to async/await.

View File

@@ -1 +0,0 @@
Correct the parameters of a test fixture. Contributed by Isaiah Singletary.

View File

@@ -1 +0,0 @@
Add documentation to the `password_providers` config option. Add known password provider implementations to docs.

View File

@@ -1 +0,0 @@
Fix --help command-line argument.

View File

@@ -1 +0,0 @@
Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev.

View File

@@ -1 +0,0 @@
Fix a bug that prevented cross-signing with users on worker-mode synapses.

1
changelog.d/7256.feature Normal file
View File

@@ -0,0 +1 @@
Add OpenID Connect login/registration support. Contributed by Quentin Gliech, on behalf of [les Connecteurs](https://connecteu.rs).

View File

@@ -1 +0,0 @@
Do not allow a deactivated user to login via SSO.

View File

@@ -1 +0,0 @@
Fix room publish permissions not being checked on room creation.

View File

@@ -1 +0,0 @@
Convert auth handler to async/await.

View File

@@ -1 +0,0 @@
Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews.

View File

@@ -1 +0,0 @@
Fix a sql query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log.

1
changelog.d/7281.misc Normal file
View File

@@ -0,0 +1 @@
Add MultiWriterIdGenerator to support multiple concurrent writers of streams.

1
changelog.d/7317.feature Normal file
View File

@@ -0,0 +1 @@
Add room details admin endpoint. Contributed by Awesome Technologies Innovationslabor GmbH.

View File

@@ -1 +0,0 @@
Move catchup of replication streams logic to worker.

1
changelog.d/7382.misc Normal file
View File

@@ -0,0 +1 @@
Add typing annotations in `synapse.federation`.

1
changelog.d/7396.misc Normal file
View File

@@ -0,0 +1 @@
Convert the room handler to async/await.

1
changelog.d/7398.docker Normal file
View File

@@ -0,0 +1 @@
Update docker runtime image to Alpine v3.11. Contributed by @Starbix.

1
changelog.d/7428.misc Normal file
View File

@@ -0,0 +1 @@
Improve performance of `get_e2e_cross_signing_key`.

1
changelog.d/7429.misc Normal file
View File

@@ -0,0 +1 @@
Improve performance of `mark_as_sent_devices_by_remote`.

1
changelog.d/7435.feature Normal file
View File

@@ -0,0 +1 @@
Allow for using more than one spam checker module at once.

1
changelog.d/7436.misc Normal file
View File

@@ -0,0 +1 @@
Support any process writing to cache invalidation stream.

1
changelog.d/7440.misc Normal file
View File

@@ -0,0 +1 @@
Refactor event persistence database functions in preparation for allowing them to be run on non-master processes.

1
changelog.d/7445.misc Normal file
View File

@@ -0,0 +1 @@
Add type hints to the SAML handler.

Some files were not shown because too many files have changed in this diff Show More