kaslo/synapse
1
0
Fork 0
Code Wiki Activity

Merge commit 'f12589547' into anoa/dinsic_release_1_23_1

Browse Source
This commit is contained in:
Andrew Morgan
2020-12-31 14:55:18 +00:00
parent 3dd861e6aa f125895475
commit 917d0f9250
23 changed files with 112 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CHANGES.md
View File

@@ -5,10 +5,10 @@ Features
--------
- Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](https://github.com/matrix-org/synapse/issues/8286))
- Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](https://github.com/matrix-org/synapse/issues/8519))
- Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](https://github.com/matrix-org/synapse/issues/8519))
- Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](https://github.com/matrix-org/synapse/issues/8539))
- Support generating structured logs via the standard logging configuration. ([\#8607](https://github.com/matrix-org/synapse/issues/8607), [\#8685](https://github.com/matrix-org/synapse/issues/8685))
- Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](https://github.com/matrix-org/synapse/issues/8610), [\#8689](https://github.com/matrix-org/synapse/issues/8689))
- Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](https://github.com/matrix-org/synapse/issues/8610), [\#8689](https://github.com/matrix-org/synapse/issues/8689))
- Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](https://github.com/matrix-org/synapse/issues/8647))
- Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](https://github.com/matrix-org/synapse/issues/8700))
- Add `displayname` to Shared-Secret Registration for admins. ([\#8722](https://github.com/matrix-org/synapse/issues/8722))
@@ -25,11 +25,11 @@ Bugfixes
- Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](https://github.com/matrix-org/synapse/issues/8632))
- Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](https://github.com/matrix-org/synapse/issues/8643))
- Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](https://github.com/matrix-org/synapse/issues/8682))
- Fix bug where Synapse would not recover after losing connection to the database. ([\#8726](https://github.com/matrix-org/synapse/issues/8726))
- Fix bug that prevented Synapse from recovering after losing connection to the database. ([\#8726](https://github.com/matrix-org/synapse/issues/8726))
- Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](https://github.com/matrix-org/synapse/issues/8728))
- Fix port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](https://github.com/matrix-org/synapse/issues/8729))
- Fix port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](https://github.com/matrix-org/synapse/issues/8730))
- Fix port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](https://github.com/matrix-org/synapse/issues/8755))
- Fix PostgreSQL port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](https://github.com/matrix-org/synapse/issues/8729))
- Fix PostgreSQL port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](https://github.com/matrix-org/synapse/issues/8730))
- Fix PostgreSQL port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](https://github.com/matrix-org/synapse/issues/8755))
Improved Documentation

1
changelog.d/8734.doc Normal file
View File

@@ -0,0 +1 @@
Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.

1
changelog.d/8751.misc Normal file
View File

@@ -0,0 +1 @@
Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite.

1
changelog.d/8754.misc Normal file
View File

@@ -0,0 +1 @@
Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.

1
changelog.d/8758.misc Normal file
View File

@@ -0,0 +1 @@
Refactor test utilities for injecting HTTP requests.

1
changelog.d/8759.misc Normal file
View File

@@ -0,0 +1 @@
Refactor test utilities for injecting HTTP requests.

1
changelog.d/8760.misc Normal file
View File

@@ -0,0 +1 @@
Refactor test utilities for injecting HTTP requests.

5
docs/sample_config.yaml
View File

@@ -1317,8 +1317,9 @@ url_preview_accept_language:
# email will be globally disabled.
#
# Additionally, if `msisdn` is not set, registration and password resets via msisdn
# will be disabled regardless. This is due to Synapse currently not supporting any
# method of sending SMS messages on its own.
# will be disabled regardless, and users will not be able to associate an msisdn
# identifier to their account. This is due to Synapse currently not supporting
# any method of sending SMS messages on its own.
#
# To enable using an identity server for operations regarding a particular third-party
# identifier type, set the value to the URL of that identity server as shown in the

5
synapse/config/registration.py
View File

@@ -322,8 +322,9 @@ class RegistrationConfig(Config):
# email will be globally disabled.
#
# Additionally, if `msisdn` is not set, registration and password resets via msisdn
# will be disabled regardless. This is due to Synapse currently not supporting any
# method of sending SMS messages on its own.
# will be disabled regardless, and users will not be able to associate an msisdn
# identifier to their account. This is due to Synapse currently not supporting
# any method of sending SMS messages on its own.
#
# To enable using an identity server for operations regarding a particular third-party
# identifier type, set the value to the URL of that identity server as shown in the

10
synapse/handlers/federation.py
View File

@@ -67,7 +67,7 @@ from synapse.replication.http.devices import ReplicationUserDevicesResyncRestSer
from synapse.replication.http.federation import (
ReplicationCleanRoomRestServlet,
ReplicationFederationSendEventsRestServlet,
ReplicationStoreRoomOnInviteRestServlet,
ReplicationStoreRoomOnOutlierMembershipRestServlet,
)
from synapse.state import StateResolutionStore
from synapse.storage.databases.main.events_worker import EventRedactBehaviour
@@ -152,12 +152,14 @@ class FederationHandler(BaseHandler):
self._user_device_resync = ReplicationUserDevicesResyncRestServlet.make_client(
hs
)
self._maybe_store_room_on_invite = ReplicationStoreRoomOnInviteRestServlet.make_client(
self._maybe_store_room_on_outlier_membership = ReplicationStoreRoomOnOutlierMembershipRestServlet.make_client(
hs
)
else:
self._device_list_updater = hs.get_device_handler().device_list_updater
self._maybe_store_room_on_invite = self.store.maybe_store_room_on_invite
self._maybe_store_room_on_outlier_membership = (
self.store.maybe_store_room_on_outlier_membership
)
# When joining a room we need to queue any events for that room up.
# For each room, a list of (pdu, origin) tuples.
@@ -1627,7 +1629,7 @@ class FederationHandler(BaseHandler):
# keep a record of the room version, if we don't yet know it.
# (this may get overwritten if we later get a different room version in a
# join dance).
await self._maybe_store_room_on_invite(
await self._maybe_store_room_on_outlier_membership(
room_id=event.room_id, room_version=room_version
)

36
synapse/handlers/room_member.py
View File

@@ -1165,32 +1165,34 @@ class RoomMemberMasterHandler(RoomMemberHandler):
#
logger.warning("Failed to reject invite: %s", e)
return await self._locally_reject_invite(
return await self._generate_local_out_of_band_leave(
invite_event, txn_id, requester, content
)
async def _locally_reject_invite(
async def _generate_local_out_of_band_leave(
self,
invite_event: EventBase,
previous_membership_event: EventBase,
txn_id: Optional[str],
requester: Requester,
content: JsonDict,
) -> Tuple[str, int]:
"""Generate a local invite rejection
"""Generate a local leave event for a room
This is called after we fail to reject an invite via a remote server. It
generates an out-of-band membership event locally.
This can be called after we e.g fail to reject an invite via a remote server.
It generates an out-of-band membership event locally.
Args:
invite_event: the invite to be rejected
previous_membership_event: the previous membership event for this user
txn_id: optional transaction ID supplied by the client
requester: user making the rejection request, according to the access token
content: additional content to include in the rejection event.
requester: user making the request, according to the access token
content: additional content to include in the leave event.
Normally an empty dict.
"""
room_id = invite_event.room_id
target_user = invite_event.state_key
Returns:
A tuple containing (event_id, stream_id of the leave event)
"""
room_id = previous_membership_event.room_id
target_user = previous_membership_event.state_key
content["membership"] = Membership.LEAVE
@@ -1202,12 +1204,12 @@ class RoomMemberMasterHandler(RoomMemberHandler):
"state_key": target_user,
}
# the auth events for the new event are the same as that of the invite, plus
# the invite itself.
# the auth events for the new event are the same as that of the previous event, plus
# the event itself.
#
# the prev_events are just the invite.
prev_event_ids = [invite_event.event_id]
auth_event_ids = invite_event.auth_event_ids() + prev_event_ids
# the prev_events consist solely of the previous membership event.
prev_event_ids = [previous_membership_event.event_id]
auth_event_ids = previous_membership_event.auth_event_ids() + prev_event_ids
event, context = await self.event_creation_handler.create_event(
requester,

10
synapse/replication/http/federation.py
View File

@@ -254,20 +254,20 @@ class ReplicationCleanRoomRestServlet(ReplicationEndpoint):
return 200, {}
class ReplicationStoreRoomOnInviteRestServlet(ReplicationEndpoint):
class ReplicationStoreRoomOnOutlierMembershipRestServlet(ReplicationEndpoint):
"""Called to clean up any data in DB for a given room, ready for the
server to join the room.
Request format:
POST /_synapse/replication/store_room_on_invite/:room_id/:txn_id
POST /_synapse/replication/store_room_on_outlier_membership/:room_id/:txn_id
{
"room_version": "1",
}
"""
NAME = "store_room_on_invite"
NAME = "store_room_on_outlier_membership"
PATH_ARGS = ("room_id",)
def __init__(self, hs):
@@ -282,7 +282,7 @@ class ReplicationStoreRoomOnInviteRestServlet(ReplicationEndpoint):
async def _handle_request(self, request, room_id):
content = parse_json_object_from_request(request)
room_version = KNOWN_ROOM_VERSIONS[content["room_version"]]
await self.store.maybe_store_room_on_invite(room_id, room_version)
await self.store.maybe_store_room_on_outlier_membership(room_id, room_version)
return 200, {}
@@ -291,4 +291,4 @@ def register_servlets(hs, http_server):
ReplicationFederationSendEduRestServlet(hs).register(http_server)
ReplicationGetQueryRestServlet(hs).register(http_server)
ReplicationCleanRoomRestServlet(hs).register(http_server)
ReplicationStoreRoomOnInviteRestServlet(hs).register(http_server)
ReplicationStoreRoomOnOutlierMembershipRestServlet(hs).register(http_server)

10
synapse/storage/databases/main/room.py
View File

@@ -1262,13 +1262,15 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
logger.error("store_room with room_id=%s failed: %s", room_id, e)
raise StoreError(500, "Problem creating room.")
async def maybe_store_room_on_invite(self, room_id: str, room_version: RoomVersion):
async def maybe_store_room_on_outlier_membership(
self, room_id: str, room_version: RoomVersion
):
"""
When we receive an invite over federation, store the version of the room if we
don't already know the room version.
When we receive an invite or any other event over federation that may relate to a room
we are not in, store the version of the room if we don't already know the room version.
"""
await self.db_pool.simple_upsert(
desc="maybe_store_room_on_invite",
desc="maybe_store_room_on_outlier_membership",
table="rooms",
keyvalues={"room_id": room_id},
values={},

4
tests/replication/_base.py
View File

@@ -240,8 +240,8 @@ class BaseMultiWorkerStreamTestCase(unittest.HomeserverTestCase):
lambda: self._handle_http_replication_attempt(self.hs, 8765),
)
def create_test_json_resource(self):
"""Overrides `HomeserverTestCase.create_test_json_resource`.
def create_test_resource(self):
"""Overrides `HomeserverTestCase.create_test_resource`.
"""
# We override this so that it automatically registers all the HTTP
# replication servlets, without having to explicitly do that in all

2
tests/rest/admin/test_admin.py
View File

@@ -35,7 +35,7 @@ from tests import unittest
class VersionTestCase(unittest.HomeserverTestCase):
url = "/_synapse/admin/v1/server_version"
def create_test_json_resource(self):
def create_test_resource(self):
resource = JsonResource(self.hs)
VersionServlet(self.hs).register(resource)
return resource

4
tests/rest/client/v1/utils.py
View File

@@ -317,9 +317,7 @@ class RestHelper:
path,
content=image_data,
access_token=tok,
)
request.requestHeaders.addRawHeader(
b"Content-Length", str(image_length).encode("UTF-8")
custom_headers=[(b"Content-Length", str(image_length))],
)
request.render(resource)
self.hs.get_reactor().pump([100])

8
tests/rest/key/v2/test_remote_key_resource.py
View File

@@ -32,7 +32,7 @@ from synapse.util.httpresourcetree import create_resource_tree
from synapse.util.stringutils import random_string
from tests import unittest
from tests.server import FakeChannel, wait_until_result
from tests.server import FakeChannel
from tests.utils import default_config
@@ -41,7 +41,7 @@ class BaseRemoteKeyResourceTestCase(unittest.HomeserverTestCase):
self.http_client = Mock()
return self.setup_test_homeserver(http_client=self.http_client)
def create_test_json_resource(self):
def create_test_resource(self):
return create_resource_tree(
{"/_matrix/key/v2": KeyApiV2Resource(self.hs)}, root_resource=NoResource()
)
@@ -94,7 +94,7 @@ class RemoteKeyResourceTestCase(BaseRemoteKeyResourceTestCase):
% (server_name.encode("utf-8"), key_id.encode("utf-8")),
b"1.1",
)
wait_until_result(self.reactor, req)
channel.await_result()
self.assertEqual(channel.code, 200)
resp = channel.json_body
return resp
@@ -190,7 +190,7 @@ class EndToEndPerspectivesTests(BaseRemoteKeyResourceTestCase):
req.requestReceived(
b"POST", path.encode("utf-8"), b"1.1",
)
wait_until_result(self.reactor, req)
channel.await_result()
self.assertEqual(channel.code, 200)
resp = channel.json_body
return resp

6
tests/rest/test_health.py
View File

@@ -20,11 +20,9 @@ from tests import unittest
class HealthCheckTests(unittest.HomeserverTestCase):
def setUp(self):
super().setUp()
def create_test_resource(self):
# replace the JsonResource with a HealthResource.
self.resource = HealthResource()
return HealthResource()
def test_health(self):
request, channel = self.make_request("GET", "/health", shorthand=False)

6
tests/rest/test_well_known.py
View File

@@ -20,11 +20,9 @@ from tests import unittest
class WellKnownTests(unittest.HomeserverTestCase):
def setUp(self):
super().setUp()
def create_test_resource(self):
# replace the JsonResource with a WellKnownResource
self.resource = WellKnownResource(self.hs)
return WellKnownResource(self.hs)
def test_well_known(self):
self.hs.config.public_baseurl = "https://tesths"

1
tests/rulecheck/test_domainrulecheck.py
View File

@@ -325,6 +325,7 @@ class DomainRuleCheckerRoomTestCase(unittest.HomeserverTestCase):
request, channel = make_request(
self.hs.get_reactor(),
self.site,
"POST",
path,
content=json.dumps(content).encode("utf8"),

53
tests/server.py
View File

@@ -2,7 +2,7 @@ import json
import logging
from collections import deque
from io import SEEK_END, BytesIO
from typing import Callable
from typing import Callable, Iterable, Optional, Tuple, Union
import attr
from typing_extensions import Deque
@@ -118,6 +118,25 @@ class FakeChannel:
def transport(self):
return self
def await_result(self, timeout: int = 100) -> None:
"""
Wait until the request is finished.
"""
self._reactor.run()
x = 0
while not self.result.get("done"):
# If there's a producer, tell it to resume producing so we get content
if self._producer:
self._producer.resumeProducing()
x += 1
if x > timeout:
raise TimedOutException("Timed out waiting for request to finish.")
self._reactor.advance(0.1)
class FakeSite:
"""
@@ -152,6 +171,9 @@ def make_request(
shorthand=True,
federation_auth_origin=None,
content_is_form=False,
custom_headers: Optional[
Iterable[Tuple[Union[bytes, str], Union[bytes, str]]]
] = None,
):
"""
Make a web request using the given method and path, feed it the
@@ -172,6 +194,8 @@ def make_request(
content_is_form: Whether the content is URL encoded form data. Adds the
'Content-Type': 'application/x-www-form-urlencoded' header.
custom_headers: (name, value) pairs to add as request headers
Returns:
Tuple[synapse.http.site.SynapseRequest, channel]
"""
@@ -225,35 +249,18 @@ def make_request(
# Assume the body is JSON
req.requestHeaders.addRawHeader(b"Content-Type", b"application/json")
if custom_headers:
for k, v in custom_headers:
req.requestHeaders.addRawHeader(k, v)
req.requestReceived(method, path, b"1.1")
return req, channel
def wait_until_result(clock, request, timeout=100):
"""
Wait until the request is finished.
"""
clock.run()
x = 0
while not request.finished:
# If there's a producer, tell it to resume producing so we get content
if request._channel._producer:
request._channel._producer.resumeProducing()
x += 1
if x > timeout:
raise TimedOutException("Timed out waiting for request to finish.")
clock.advance(0.1)
def render(request, resource, clock):
request.render(resource)
wait_until_result(clock, request)
request._channel.await_result()
@implementer(IReactorPluggableNameResolver)

14
tests/storage/test_client_ips.py
View File

@@ -21,6 +21,7 @@ from synapse.http.site import XForwardedForRequest
from synapse.rest.client.v1 import login
from tests import unittest
from tests.server import make_request
from tests.test_utils import make_awaitable
from tests.unittest import override_config
@@ -408,17 +409,18 @@ class ClientIpAuthTestCase(unittest.HomeserverTestCase):
# Advance to a known time
self.reactor.advance(123456 - self.reactor.seconds())
request, channel = self.make_request(
headers1 = {b"User-Agent": b"Mozzila pizza"}
headers1.update(headers)
request, channel = make_request(
self.reactor,
self.site,
"GET",
"/_matrix/client/r0/admin/users/" + self.user_id,
access_token=access_token,
custom_headers=headers1.items(),
**make_request_args,
)
request.requestHeaders.addRawHeader(b"User-Agent", b"Mozzila pizza")
# Add the optional headers
for h, v in headers.items():
request.requestHeaders.addRawHeader(h, v)
self.render(request)
# Advance so the save loop occurs

18
tests/unittest.py
View File

@@ -30,6 +30,7 @@ from twisted.internet.defer import Deferred, ensureDeferred, succeed
from twisted.python.failure import Failure
from twisted.python.threadpool import ThreadPool
from twisted.trial import unittest
from twisted.web.resource import Resource
from synapse.api.constants import EventTypes, Membership
from synapse.config.homeserver import HomeServerConfig
@@ -239,10 +240,8 @@ class HomeserverTestCase(TestCase):
if not isinstance(self.hs, HomeServer):
raise Exception("A homeserver wasn't returned, but %r" % (self.hs,))
# Register the resources
self.resource = self.create_test_json_resource()
# create a site to wrap the resource.
# create the root resource, and a site to wrap it.
self.resource = self.create_test_resource()
self.site = SynapseSite(
logger_name="synapse.access.http.fake",
site_tag=self.hs.config.server.server_name,
@@ -323,15 +322,12 @@ class HomeserverTestCase(TestCase):
hs = self.setup_test_homeserver()
return hs
def create_test_json_resource(self):
def create_test_resource(self) -> Resource:
"""
Create a test JsonResource, with the relevant servlets registerd to it
Create a the root resource for the test server.
The default implementation calls each function in `servlets` to do the
registration.
Returns:
JsonResource:
The default implementation creates a JsonResource and calls each function in
`servlets` to register servletes against it
"""
resource = JsonResource(self.hs)