1
0
Commit Graph

19974 Commits

Author SHA1 Message Date
Olivier Wilkinson (reivilibre)
eb5c1e2075 Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2021-12-29 11:37:10 +00:00
Olivier Wilkinson (reivilibre)
5d19fa6ffe Add a test for sending OTKs and UFBKs to ASes upon receiving PDUs 2021-12-29 11:37:09 +00:00
Olivier Wilkinson (reivilibre)
4306641df1 Allow passing an appservice_user_id to the join helper 2021-12-29 11:37:09 +00:00
Olivier Wilkinson (reivilibre)
a8a64fdd19 Return the device ID when registering an appservice user in test helpers 2021-12-29 11:37:09 +00:00
Olivier Wilkinson (reivilibre)
2905fcfad2 Fix the get_bulk_e2e_unused_fallback_keys query to return devices with only used keys 2021-12-29 11:37:09 +00:00
Olivier Wilkinson (reivilibre)
16803645c8 Break up get_app_service_users_in_room to make it easier to debug 2021-12-29 11:37:08 +00:00
Olivier Wilkinson (reivilibre)
fe185a11b8 Get unused fallback key types in bulk and send them out 2021-12-29 11:37:08 +00:00
Olivier Wilkinson (reivilibre)
3ca3f34b74 Count the OTKs in bulk 2021-12-29 11:37:08 +00:00
Olivier Wilkinson (reivilibre)
9661526a30 Fix up tests that weren't expecting extra call arguments 2021-12-29 11:36:31 +00:00
Olivier Wilkinson (reivilibre)
1bac476ba4 Fix two tests 2021-12-29 11:36:31 +00:00
Olivier Wilkinson (reivilibre)
dc7e6c1cfc Find interesting users for the AS when sending OTKs and FBKs 2021-12-29 11:36:30 +00:00
Olivier Wilkinson (reivilibre)
71dc5e94b0 Add some method stubs and add the OTKs and FBKs to the response 2021-12-29 11:35:27 +00:00
Olivier Wilkinson (reivilibre)
4cba87c025 Pipe through the feature flag 2021-12-29 11:35:01 +00:00
Olivier Wilkinson (reivilibre)
c5d7988ad0 Add feature flag for experimental MSC3202 transaction extensions 2021-12-29 11:32:59 +00:00
Olivier Wilkinson (reivilibre)
a10a41f639 Fix up some tests 2021-12-29 11:32:59 +00:00
Olivier Wilkinson (reivilibre)
3e0a9f4f11 Boring piping 2021-12-29 11:32:58 +00:00
Olivier Wilkinson (reivilibre)
d368d37061 Fix up type after rebase onto anoa's branch 2021-12-29 11:32:21 +00:00
Olivier Wilkinson (reivilibre)
474ff3b9fa During AS catch-up, send empty OTK counts and fallback keys for now 2021-12-29 11:31:50 +00:00
Olivier Wilkinson (reivilibre)
307afbcf57 Emit the one-time key counts and fallback keys over federation 2021-12-29 11:31:50 +00:00
Olivier Wilkinson (reivilibre)
bd3dfafa3d Feed one-time key counts and unused fallback keys through the transaction 2021-12-29 11:31:50 +00:00
Olivier Wilkinson (reivilibre)
20076d93a6 Add type aliases for one-time key counts and unused fallback keys that will be sent 2021-12-29 11:30:32 +00:00
Olivier Wilkinson (reivilibre)
bf4488336f Support opting-in to MSC3202 transactional behaviour using the registration file 2021-12-29 11:30:31 +00:00
Andrew Morgan
2856d73830 Fix calls to create_appservice_txn in tests 2021-12-29 11:29:51 +00:00
Andrew Morgan
c16ddc1df9 Modify tests to handle new location of to-device messages in AS txns 2021-12-29 11:29:51 +00:00
Andrew Morgan
74d29c1fd7 Insert to-device messages into the new to-device part of AS txns 2021-12-29 11:29:51 +00:00
Andrew Morgan
1447897ac8 Add to-device messages as their own special section in AS txns 2021-12-29 11:29:05 +00:00
Andrew Morgan
6cc14262b4 Fix tests to mock _TransactionController.send of ApplicationServiceScheduler.enqueue*
With enqueue_for_appservice being called per-event per-appservice, it's
not a great target for mocking. So we use _TransactionController.send
instead, to track things that are actually sent out to AS's.

This also has the benefit of testing a wider bit of the AS txn pipeline
2021-12-29 11:27:05 +00:00
Andrew Morgan
8375272fa3 Refactor and generalise the sending of arbitrary fields over AS transactions
Things were starting to get a little inflexible as we kept adding new
types of data to send to application services. It's better to just
have one method for adding data to AS transactions, than one for
each type of data.

Note that subsequent PRs will need to add device lists, one-time keys
and fallback keys to these transactions. Adding those are additional
arguments to a method is much nicer than a new method for each one.

Plus with this setup we can add multiple different types of data at
once without kicking off an AS transaction for each type. This will
be useful for OTK/fallback keys, as we plan to lazily attach those
when handling other event types.
2021-12-29 11:27:04 +00:00
Andrew Morgan
49bbf65796 Add comment on why we don't NOT NULL to_device_stream_id 2021-12-29 11:23:10 +00:00
Andrew Morgan
6f7792a6ee Add some FIXME comments 2021-12-29 11:23:10 +00:00
Andrew Morgan
50b3cf7755 rename set_type_stream_id_for_appservice -> set_appservice_stream_type_pos 2021-12-29 11:23:10 +00:00
Andrew Morgan
124bb71442 Fix existing unit tests
There is so much mocking going on here. I look forward to replacing these one day.
2021-12-29 11:22:40 +00:00
Andrew Morgan
75b8bd3951 Simplify registration of appservices in tests 2021-12-29 11:22:40 +00:00
Andrew Morgan
a38cb2faf7 Rename user1, user2 in tests to something more useful 2021-12-29 11:22:39 +00:00
Andrew Morgan
d5d4d7832d Rename ApplicationServiceEphemeralEventsTestCase 2021-12-29 11:22:39 +00:00
Andrew Morgan
cc30393cee _handle_to_device -> _get_to_device_messages 2021-12-29 11:22:39 +00:00
Andrew Morgan
d4a93df981 Deduplicate ephemeral events to send conditional
Test cases needed to be updated, as we now always call
submit_ephemeral_events_for_as, it may just be with an
empty events list.
2021-12-29 11:22:39 +00:00
Andrew Morgan
edfb4f5e9f Broaden type hints; update comment 2021-12-29 11:22:38 +00:00
Andrew Morgan
0fe753cc7f Move stream filter back into AppserviceHandler 2021-12-29 11:21:35 +00:00
Andrew Morgan
b9b4b55092 Make msc2409_to_device_messages_enabled private; remove unnecessary check
The second check for self._msc2409_to_device_messages_enabled was not necessary. It's
already checked in notify_interested_services_ephemeral earlier.
2021-12-29 11:21:34 +00:00
Andrew Morgan
076f959a2d Changelog 2021-12-29 11:21:34 +00:00
Andrew Morgan
94a2389a97 Add tests
I decided to spin up another test class for this as the existing one is
1. quite old and 2. was mocking away too much of the infrastructure to
my liking. I've named the new class alluding to ephemeral messages, and
while we already have some ephemeral tests in AppServiceHandlerTestCase,
ideally I'd like to migrate those over.

There's two new tests here. One for testing that to-device messages for
a local user are received by any application services that have
registered interest in that user - and that those that haven't won't
receive those messages.

The next test is similar, but tests with a whole bunch of to-device
messages. Rather than actually registering tons of devices - which would
make for a very slow unit test - we just directly insert them into the
database.
2021-12-29 11:21:34 +00:00
Andrew Morgan
662756aa7f Add a to_device_stream_id column to the application_services_state table
This is for tracking the stream id that each application service has
been sent up to. In other words, there shouldn't be any need to process
stream ids below the recorded one here as the AS should have already
received them.

Note that there is no reliability built-in here. Reliability of delivery
if intended for a separate PR.
2021-12-29 11:21:34 +00:00
Andrew Morgan
d6288fff7e Add database method to fetch to-device messages by user_ids from db
This method is quite similar to the one below, except that it doesn't
support device ids, and supports querying with more than one user id,
both of which are relevant to application services.

The results are also formatted in a different data structure, so I'm
not sure how much we could really share here between the two methods.
2021-12-29 11:21:33 +00:00
Andrew Morgan
668ac822de Allow setting/getting stream id per appservice for to-device messages 2021-12-29 11:21:33 +00:00
Andrew Morgan
a1ac23a2b6 Add a new ephemeral AS handler for to_device message edus
Here we add the ability for the application service ephemeral message
processor to handle new events on the "to_device" stream.

We keep track of a stream id (token) per application service, and every
time a new to-device message comes in, for each appservice we pull the
messages between the last-recorded and current stream id and check
whether any of the messages are for a user in that appservice's user
namespace.

get_new_messages is implemented in the next commit.

since we rebased off latest develop.
2021-12-29 11:21:33 +00:00
Andrew Morgan
31078e10a1 Add experimental config option to send to-device messages to AS's 2021-12-29 11:21:33 +00:00
Shay
fcfe67578f Update to the current version of Black and run it on Synapse codebase (#11596)
* update black version

* run updated version of black on code

* newsfragment

* enumerate python versions
2021-12-23 20:22:15 -08:00
Patrick Cloke
66d7aa783a Fix mypy error with opentracing.tags. (#11622) 2021-12-23 11:47:24 +00:00
AndrewFerr
c500bf37d6 Add details for how to set up TURN behind NAT (#11553)
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-12-23 10:42:50 +00:00