1
0
Commit Graph

18386 Commits

Author SHA1 Message Date
Richard van der Hoff 8d6f97f932 Merge remote-tracking branch 'origin/release-v1.20.0' into develop 2020-09-08 09:58:07 +01:00
Brendan Abolivier d8762cc116 Only add rows to the push actions table if the event notifies or should be marked unread (#8274) 2020-09-07 16:56:27 +01:00
Richard van der Hoff 77794ebc77 Fix stack overflow when logging system encounters an error (#8268) 2020-09-07 16:54:30 +01:00
Richard van der Hoff ef2804d27c Avoid table-scanning users at startup (#8271)
This takes about 10 seconds in the best case; often more.
2020-09-07 16:48:52 +01:00
Brendan Abolivier a55e2707d7 Fix unread count failing on NULL values (#8270)
Fix unread counts making sync fail if the value of the `unread_count`
column in `event_push_summary` is `None`.
2020-09-07 15:15:06 +01:00
Richard van der Hoff 0dae7d80bf Add more logging to debug slow startup (#8264)
I'm hoping this will provide some pointers for debugging
https://github.com/matrix-org/synapse/issues/7968.
2020-09-07 13:36:02 +01:00
Richard van der Hoff 96312536f2 Refuse to upgrade database on worker processes (#8266) 2020-09-07 13:04:10 +01:00
Alexandre Morignot 7586fdf1e8 Bump canonicaljson to version 1.4.0 (#8262)
The version 1.3.0 has a bug with unicode charecters:
```
>>> from canonicaljson import encode_pretty_printed_json
>>> encode_pretty_printed_json({'a': 'à'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/erdnaxeli/.pyenv/versions/3.6.7/lib/python3.6/site-packages/canonicaljson.py", line 96, in encode_pretty_printed_json
    return _pretty_encoder.encode(json_object).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character '\xe0' in position 12: ordinal not in range(128)
```

Signed-off-by: Alexandre Morignot <erdnaxeli@cervoi.se>

Co-authored-by: Alexandre Morignot <erdnaxeli@cervoi.se>
2020-09-07 12:21:38 +01:00
Richard van der Hoff 5b452df23b Run database updates in a transaction (#8265)
Fixes: #6467
2020-09-07 11:41:50 +01:00
reivilibre 765437df54 Add tests for last_successful_stream_ordering (#8258) 2020-09-07 10:11:38 +01:00
Richard van der Hoff 77b4711bc2 Merge branch 'release-v1.20.0' into develop 2020-09-06 23:32:28 +01:00
reivilibre 7513006b09 In light of #8255, use BIGINTs for destination_rooms (#8256) 2020-09-04 15:07:29 +01:00
reivilibre 17fa4c7ca7 Catch up after Federation Outage (split, 2): Track last successful stream ordering after transmission (#8247)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-09-04 15:06:51 +01:00
Richard van der Hoff f25af1f9c7 Add cross-signing sigs to the keys object (#8234)
All the callers want this info in the same place, so let's reduce the
duplication by doing it here.
2020-09-04 15:06:05 +01:00
Brendan Abolivier 041ee971c9 Unread counts fixes (#8254)
* Fixup `ALTER TABLE` database queries

Make the new columns nullable, because doing otherwise can wedge a
server with a big database, as setting a default value rewrites the
table.

* Switch back to using the notifications count in the push badge

Clients are likely to be confused if we send a push but the badge count
is the unread messages one, and not the notifications one.

* Changelog
2020-09-04 14:14:22 +01:00
Patrick Cloke db7de4d182 Fix a regression from calling read_templates. (#8252)
Regressed in #8037.
2020-09-04 09:10:33 -04:00
reivilibre 58f61f10f7 Catch-up after Federation Outage (split, 1) (#8230)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2020-09-04 12:22:23 +01:00
reivilibre e351298444 Fix type signature in simple_select_one_onecol and friends (#8241)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-09-04 12:02:29 +01:00
Patrick Cloke c619253db8 Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Brendan Abolivier 9f8abdcc38 Revert "Add experimental support for sharding event persister. (#8170)" (#8242)
* Revert "Add experimental support for sharding event persister. (#8170)"

This reverts commit 82c1ee1c22.

* Changelog
2020-09-04 10:19:42 +01:00
Erik Johnston be16ee59a8 Add type hints to more handlers (#8244) 2020-09-03 22:02:29 +01:00
reivilibre 4535e849d7 Remove obsolete order field in send_new_transaction (#8245)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-09-03 19:23:07 +01:00
Richard van der Hoff f97f9485ee Split fetching device keys and signatures into two transactions (#8233)
I think this is simpler (and moves stuff out of the db threads)
2020-09-03 18:27:26 +01:00
Erik Johnston 208e1d3eb3 Fix typing for @cached wrapped functions (#8240)
This requires adding a mypy plugin to fiddle with the type signatures a bit.
2020-09-03 15:38:32 +01:00
Patrick Cloke 15c35c250c Remove useless changelog about reverting a #8239. 2020-09-03 09:47:41 -04:00
Patrick Cloke 2aa127c207 Revert pinning of setuptools (#8239) 2020-09-03 09:45:36 -04:00
Erik Johnston 5bfc79486d Fix typing for SyncHandler (#8237) 2020-09-03 12:54:10 +01:00
Richard van der Hoff 6f6f371a87 wrap _get_e2e_device_keys_and_signatures_txn in a non-txn method (#8231)
We have three things which all call `_get_e2e_device_keys_and_signatures_txn`
with their own `runInteraction`. Factor out the common code.
2020-09-03 11:50:49 +01:00
Patrick Cloke c8758cb72f Add an overload for simple_select_one_onecol_txn. (#8235) 2020-09-02 15:03:12 -04:00
Patrick Cloke 912e024913 Convert runInteraction to async/await (#8156) 2020-09-02 13:11:02 -04:00
Erik Johnston 112266eafd Add StreamStore to mypy (#8232) 2020-09-02 17:52:38 +01:00
Brendan Abolivier 5a1dd297c3 Re-implement unread counts (again) (#8059) 2020-09-02 17:19:37 +01:00
Richard van der Hoff 0d4f614fda Refactor _get_e2e_device_keys_for_federation_query_txn (#8225)
We can use the existing `_get_e2e_device_keys_and_signatures_txn` instead of
creating our own txn function
2020-09-02 15:53:26 +01:00
Erik Johnston 82c1ee1c22 Add experimental support for sharding event persister. (#8170)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
Will Hunt b257c788c0 Add /user/{user_id}/shared_rooms/ api (#7785)
* Add shared_rooms api

* Add changelog

* Add .

* Wrap response in {"rooms": }

* linting

* Add unstable_features key

* Remove options from isort that aren't part of 5.x

`-y` and `-rc` are now default behaviour and no longer exist.

`dont-skip` is no longer required

https://timothycrosley.github.io/isort/CHANGELOG/#500-penny-july-4-2020

* Update imports to make isort happy

* Add changelog

* Update tox.ini file with correct invocation

* fix linting again for isort

* Vendor prefix unstable API

* Fix to match spec

* import Codes

* import Codes

* Use FORBIDDEN

* Update changelog.d/7785.feature

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Implement get_shared_rooms_for_users

* a comma

* trailing whitespace

* Handle the easy feedback

* Switch to using runInteraction

* Add tests

* Feedback

* Seperate unstable endpoint from v2

* Add upgrade node

* a line

* Fix style by adding a blank line at EOF.

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Tulir Asokan <tulir@maunium.net>

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Update UPGRADE.rst

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Fix UPGRADE/CHANGELOG unstable paths

unstable unstable unstable

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>
2020-09-02 13:18:40 +01:00
Patrick Cloke 9356656e67 Do not try to store invalid data in the stats table (#8226) 2020-09-02 07:59:39 -04:00
Patrick Cloke d250521cf5 Convert the main methods run by the reactor to async. (#8213) 2020-09-02 07:44:50 -04:00
Richard van der Hoff abeab964d5 Make _get_e2e_device_keys_and_signatures_txn return an attrs (#8224)
this makes it a bit clearer what's going on.
2020-09-02 11:47:26 +01:00
Patrick Cloke b939251c37 Fix errors when updating the user directory with invalid data (#8223) 2020-09-01 13:02:41 -04:00
Brendan Abolivier b5133dd97f Explain better what GDPR-erased means (#8189)
Fixes https://github.com/matrix-org/synapse/issues/8185
2020-09-01 16:31:59 +01:00
Patrick Cloke 37db6252b7 Convert additional databases to async/await part 3 (#8201) 2020-09-01 11:04:17 -04:00
Patrick Cloke 7d103a594e Convert appservice code to async/await. (#8207) 2020-09-01 11:03:49 -04:00
Richard van der Hoff 5615eb5cb4 Rename _get_e2e_device_keys_txn (#8222)
... to `_get_e2e_device_keys_and_signatures_txn`, to better reflect what it
does.
2020-09-01 16:02:17 +01:00
Patrick Cloke 54f8d73c00 Convert additional databases to async/await (#8199) 2020-09-01 09:21:48 -04:00
Patrick Cloke 5bf8e5f55b Convert the well known resolver to async (#8214) 2020-09-01 09:15:22 -04:00
Patrick Cloke da77520cd1 Convert additional databases to async/await part 2 (#8200) 2020-09-01 08:39:04 -04:00
Erik Johnston bbb3c8641c Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203)
This is so that we can use it for the backfill events stream.
2020-09-01 13:36:25 +01:00
Patrick Cloke 318245eaa6 Do not install setuptools 50.0. (#8212)
This is due to compatibility issues with old Python versions.
2020-09-01 08:16:58 -04:00
Richard van der Hoff aa07c37cf0 Move and rename get_devices_with_keys_by_user (#8204)
* Move `get_devices_with_keys_by_user` to `EndToEndKeyWorkerStore`

this seems a better fit for it.

This commit simply moves the existing code: no other changes at all.

* Rename `get_devices_with_keys_by_user`

to better reflect what it does.

* get_device_stream_token abstract method

To avoid referencing fields which are declared in the derived classes, make
`get_device_stream_token` abstract, and define that in the classes which define
`_device_list_id_gen`.
2020-09-01 12:41:21 +01:00
Richard van der Hoff 45e8f7726f Rename get_e2e_device_keys to better reflect its purpose (#8205)
... and to show that it does something slightly different to
`_get_e2e_device_keys_txn`.

`include_all_devices` and `include_deleted_devices` were never used (and
`include_deleted_devices` was broken, since that would cause `None`s in the
result which were not handled in the loop below.

Add some typing too.
2020-08-29 00:14:17 +01:00