kaslo/synapse
1
0
Fork 0
Code Wiki Activity

Merge branch 'develop' into matrix-org-hotfixes

Browse Source
This commit is contained in:
Richard van der Hoff
2020-07-24 09:57:49 +01:00
parent 25880bd441 1ec688bf21
commit be777e325d
277 changed files with 5677 additions and 2756 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGES.md
+64
View File
@@ -1,3 +1,67 @@
Synapse 1.17.0 (2020-07-13)
===========================
Synapse 1.17.0 is identical to 1.17.0rc1, with the addition of the fix that was included in 1.16.1.
Synapse 1.16.1 (2020-07-10)
===========================
In some distributions of Synapse 1.16.0, we incorrectly included a database migration which added a new, unused table. This release removes the redundant table.
Bugfixes
--------
- Drop table `local_rejections_stream` which was incorrectly added in Synapse 1.16.0. ([\#7816](https://github.com/matrix-org/synapse/issues/7816), [b1beb3ff5](https://github.com/matrix-org/synapse/commit/b1beb3ff5))
Synapse 1.17.0rc1 (2020-07-09)
==============================
Bugfixes
--------
- Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel. ([\#7021](https://github.com/matrix-org/synapse/issues/7021))
- Fix "Tried to close a non-active scope!" error messages when opentracing is enabled. ([\#7732](https://github.com/matrix-org/synapse/issues/7732))
- Fix incorrect error message when database CTYPE was set incorrectly. ([\#7760](https://github.com/matrix-org/synapse/issues/7760))
- Fix to not ignore `set_tweak` actions in Push Rules that have no `value`, as permitted by the specification. ([\#7766](https://github.com/matrix-org/synapse/issues/7766))
- Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian. ([\#7779](https://github.com/matrix-org/synapse/issues/7779))
- Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent. ([\#7797](https://github.com/matrix-org/synapse/issues/7797))
- Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation. ([\#7804](https://github.com/matrix-org/synapse/issues/7804), [\#7809](https://github.com/matrix-org/synapse/issues/7809), [\#7810](https://github.com/matrix-org/synapse/issues/7810))
Updates to the Docker image
---------------------------
- Include libwebp in the Docker file to properly handle webp image uploads. ([\#7791](https://github.com/matrix-org/synapse/issues/7791))
Improved Documentation
----------------------
- Improve the documentation of the non-standard JSON web token login type. ([\#7776](https://github.com/matrix-org/synapse/issues/7776))
- Update doc links for caddy. Contributed by Nicolai Søborg. ([\#7789](https://github.com/matrix-org/synapse/issues/7789))
Internal Changes
----------------
- Refactor getting replication updates from database. ([\#7740](https://github.com/matrix-org/synapse/issues/7740))
- Send push notifications with a high or low priority depending upon whether they may generate user-observable effects. ([\#7765](https://github.com/matrix-org/synapse/issues/7765))
- Use symbolic names for replication stream names. ([\#7768](https://github.com/matrix-org/synapse/issues/7768))
- Add early returns to `_check_for_soft_fail`. ([\#7769](https://github.com/matrix-org/synapse/issues/7769))
- Fix up `synapse.handlers.federation` to pass mypy. ([\#7770](https://github.com/matrix-org/synapse/issues/7770))
- Convert the appserver handler to async/await. ([\#7775](https://github.com/matrix-org/synapse/issues/7775))
- Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz. ([\#7780](https://github.com/matrix-org/synapse/issues/7780))
- Update linting scripts and codebase to be compatible with `isort` v5. ([\#7786](https://github.com/matrix-org/synapse/issues/7786))
- Stop populating unused table `local_invites`. ([\#7793](https://github.com/matrix-org/synapse/issues/7793))
- Ensure that strings (not bytes) are passed into JSON serialization. ([\#7799](https://github.com/matrix-org/synapse/issues/7799))
- Switch from simplejson to the standard library json. ([\#7800](https://github.com/matrix-org/synapse/issues/7800))
- Add `signing_key` property to `HomeServer` to save code duplication. ([\#7805](https://github.com/matrix-org/synapse/issues/7805))
- Improve stacktraces from exceptions in background processes. ([\#7808](https://github.com/matrix-org/synapse/issues/7808))
- Fix various spelling errors in comments and log lines. ([\#7811](https://github.com/matrix-org/synapse/issues/7811))
Synapse 1.16.0 (2020-07-08)
===========================
INSTALL.md
+5 -7
View File
@@ -405,13 +405,11 @@ so, you will need to edit `homeserver.yaml`, as follows:
```
* You will also need to uncomment the `tls_certificate_path` and
`tls_private_key_path` lines under the `TLS` section. You can either
point these settings at an existing certificate and key, or you can
enable Synapse's built-in ACME (Let's Encrypt) support. Instructions
for having Synapse automatically provision and renew federation
certificates through ACME can be found at [ACME.md](docs/ACME.md).
Note that, as pointed out in that document, this feature will not
work with installs set up after November 2019.
`tls_private_key_path` lines under the `TLS` section. You will need to manage
provisioning of these certificates yourself — Synapse had built-in ACME
support, but the ACMEv1 protocol Synapse implements is deprecated, not
allowed by LetsEncrypt for new sites, and will break for existing sites in
late 2020. See [ACME.md](docs/ACME.md).
If you are using your own certificate, be sure to use a `.pem` file that
includes the full certificate chain including any intermediate certificates
README.rst
+5 -5
View File
@@ -188,12 +188,8 @@ Using PostgreSQL
================
Synapse offers two database engines:
* `SQLite <https://sqlite.org/>`_
* `PostgreSQL <https://www.postgresql.org>`_
By default Synapse uses SQLite in and doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
* `SQLite <https://sqlite.org/>`_
Almost all installations should opt to use PostgreSQL. Advantages include:
@@ -207,6 +203,10 @@ Almost all installations should opt to use PostgreSQL. Advantages include:
For information on how to install and use PostgreSQL, please see
`docs/postgres.md <docs/postgres.md>`_.
By default Synapse uses SQLite and in doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
.. _reverse-proxy:
Using a reverse proxy with Synapse
changelog.d/6455.feature
+1
View File
@@ -0,0 +1 @@
Include room states on invite events that are sent to application services. Contributed by @Sorunome.
changelog.d/7021.bugfix
-1
View File
@@ -1 +0,0 @@
Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel.
changelog.d/7613.feature
+1
View File
@@ -0,0 +1 @@
Add delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel.
changelog.d/7732.bugfix
-1
View File
@@ -1 +0,0 @@
Fix "Tried to close a non-active scope!" error messages when opentracing is enabled.
changelog.d/7740.misc
-1
View File
@@ -1 +0,0 @@
Refactor getting replication updates from database.
changelog.d/7760.bugfix
-1
View File
@@ -1 +0,0 @@
Fix incorrect error message when database CTYPE was set incorrectly.
changelog.d/7765.misc
-1
View File
@@ -1 +0,0 @@
Send push notifications with a high or low priority depending upon whether they may generate user-observable effects.
changelog.d/7766.bugfix
-1
View File
@@ -1 +0,0 @@
Fix to not ignore `set_tweak` actions in Push Rules that have no `value`, as permitted by the specification.
changelog.d/7768.misc
-1
View File
@@ -1 +0,0 @@
Use symbolic names for replication stream names.
changelog.d/7769.misc
-1
View File
@@ -1 +0,0 @@
Add early returns to `_check_for_soft_fail`.
changelog.d/7770.misc
-1
View File
@@ -1 +0,0 @@
Fix up `synapse.handlers.federation` to pass mypy.
changelog.d/7775.misc
-1
View File
@@ -1 +0,0 @@
Convert the appserver handler to async/await.
changelog.d/7776.doc
-1
View File
@@ -1 +0,0 @@
Improve the documentation of the non-standard JSON web token login type.
changelog.d/7779.bugfix
-1
View File
@@ -1 +0,0 @@
Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian.
changelog.d/7780.misc
-1
View File
@@ -1 +0,0 @@
Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz.
changelog.d/7786.misc
-1
View File
@@ -1 +0,0 @@
Update linting scripts and codebase to be compatible with `isort` v5.
changelog.d/7789.doc
-1
View File
@@ -1 +0,0 @@
Update doc links for caddy. Contributed by Nicolai Søborg.
changelog.d/7791.docker
-1
View File
@@ -1 +0,0 @@
Include libwebp in the Docker file to properly handle webp image uploads.
changelog.d/7793.misc
-1
View File
@@ -1 +0,0 @@
Stop populating unused table `local_invites`.
changelog.d/7797.bugfix
-1
View File
@@ -1 +0,0 @@
Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent.
changelog.d/7798.feature
+1
View File
@@ -0,0 +1 @@
Add experimental support for running multiple federation sender processes.
changelog.d/7799.misc
-1
View File
@@ -1 +0,0 @@
Ensure that strings (not bytes) are passed into JSON serialization.
changelog.d/7800.misc
-1
View File
@@ -1 +0,0 @@
Switch from simplejson to the standard library json.
changelog.d/7802.misc
+1
View File
@@ -0,0 +1 @@
Switch from simplejson to the standard library json.
changelog.d/7804.bugfix
-1
View File
@@ -1 +0,0 @@
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation.
changelog.d/7805.misc
-1
View File
@@ -1 +0,0 @@
Add `signing_key` property to `HomeServer` to save code duplication.
changelog.d/7810.bugfix
-1
View File
@@ -1 +0,0 @@
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation.
changelog.d/7813.misc
+1
View File
@@ -0,0 +1 @@
Add type hints to the http server code and remove an unused parameter.
changelog.d/7815.bugfix
+1
View File
@@ -0,0 +1 @@
Fix detection of out of sync remote device lists when receiving events from remote users.
changelog.d/7817.bugfix
+1
View File
@@ -0,0 +1 @@
Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain.
changelog.d/7820.misc
+1
View File
@@ -0,0 +1 @@
Add type hints to synapse.api.errors module.
changelog.d/7822.bugfix
+1
View File
@@ -0,0 +1 @@
Fix a bug causing Synapse to misinterpret the value `off` for `encryption_enabled_by_default_for_room_type` in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0.
changelog.d/7827.feature
+1
View File
@@ -0,0 +1 @@
Add the option to validate the `iss` and `aud` claims for JWT logins.
changelog.d/7829.bugfix
+1
View File
@@ -0,0 +1 @@
Fix bug where we did not always pass in `app_name` or `server_name` to email templates, including e.g. for registration emails.
changelog.d/7830.feature
+1
View File
@@ -0,0 +1 @@
Add support for handling registration requests across multiple client reader workers.
changelog.d/7836.misc
+1
View File
@@ -0,0 +1 @@
Ensure that calls to `json.dumps` are compatible with the standard library json.
changelog.d/7839.docker
+1
View File
@@ -0,0 +1 @@
Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196.
changelog.d/7842.feature
+1
View File
@@ -0,0 +1 @@
Add an admin API to list the users in a room. Contributed by Awesome Technologies Innovationslabor GmbH.
changelog.d/7844.bugfix
+1
View File
@@ -0,0 +1 @@
Errors which occur while using the non-standard JWT login now return the proper error: `403 Forbidden` with an error code of `M_FORBIDDEN`.
changelog.d/7846.feature
+1
View File
@@ -0,0 +1 @@
Allow email subjects to be customised through Synapse's configuration.
changelog.d/7847.feature
+1
View File
@@ -0,0 +1 @@
Add the ability to re-activate an account from the admin API.
changelog.d/7848.misc
+1
View File
@@ -0,0 +1 @@
Remove redundant `retry_on_integrity_error` wrapper for event persistence code.
changelog.d/7849.misc
+1
View File
@@ -0,0 +1 @@
Consistently use `db_to_json` to convert from database values to JSON objects.
changelog.d/7850.bugfix
+1
View File
@@ -0,0 +1 @@
Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0.
changelog.d/7851.misc
+1
View File
@@ -0,0 +1 @@
Convert E2E keys and room keys handlers to async/await.
changelog.d/7853.misc
+1
View File
@@ -0,0 +1 @@
Add support for handling registration requests across multiple client reader workers.
changelog.d/7854.bugfix
+1
View File
@@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation.
changelog.d/7855.feature
+1
View File
@@ -0,0 +1 @@
Add experimental support for running multiple pusher workers.
changelog.d/7856.misc
+1
View File
@@ -0,0 +1 @@
Small performance improvement in typing processing.
changelog.d/7858.misc
+1
View File
@@ -0,0 +1 @@
The default value of `filter_timeline_limit` was changed from -1 (no limit) to 100.
changelog.d/7859.bugfix
+1
View File
@@ -0,0 +1 @@
Fix a bug which allowed empty rooms to be rejoined over federation.
changelog.d/7860.misc
+1
View File
@@ -0,0 +1 @@
Convert _base, profile, and _receipts handlers to async/await.
changelog.d/7861.misc
+1
View File
@@ -0,0 +1 @@
Optimise queueing of inbound replication commands.
changelog.d/7866.bugfix
+1
View File
@@ -0,0 +1 @@
Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers.
changelog.d/7868.misc
+1
View File
@@ -0,0 +1 @@
Convert synapse.app and federation client to async/await.
changelog.d/7869.feature
+1
View File
@@ -0,0 +1 @@
Add experimental support for moving typing off master.
changelog.d/7870.misc
+1
View File
@@ -0,0 +1 @@
Add some type annotations to `HomeServer` and `BaseHandler`.
changelog.d/7871.misc
+1
View File
@@ -0,0 +1 @@
Convert device handler to async/await.
changelog.d/7872.bugfix
+1
View File
@@ -0,0 +1 @@
Fix a long standing bug where the tracing of async functions with opentracing was broken.
changelog.d/7874.misc
+1
View File
@@ -0,0 +1 @@
Convert the federation agent and related code to async/await.
changelog.d/7877.misc
+1
View File
@@ -0,0 +1 @@
Clean up `PreserveLoggingContext`.
changelog.d/7878.removal
+1
View File
@@ -0,0 +1 @@
Remove unused `synapse_replication_tcp_resource_invalidate_cache` prometheus metric.
changelog.d/7879.feature
+1
View File
@@ -0,0 +1 @@
Report CPU metrics to prometheus for time spent processing replication commands.
changelog.d/7880.bugfix
+1
View File
@@ -0,0 +1 @@
Fix "TypeError in `synapse.notifier`" exceptions.
changelog.d/7881.misc
+1
View File
@@ -0,0 +1 @@
Change "unknown room version" logging from 'error' to 'warning'.
changelog.d/7882.misc
+1
View File
@@ -0,0 +1 @@
Stop using `device_max_stream_id` table and just use `device_inbox.stream_id`.
changelog.d/7884.misc
+1
View File
@@ -0,0 +1 @@
Convert the message handler to async/await.
changelog.d/7885.doc
+1
View File
@@ -0,0 +1 @@
Provide instructions on using `register_new_matrix_user` via docker.
changelog.d/7888.misc
+1
View File
@@ -0,0 +1 @@
Remove Ubuntu Eoan from the list of `.deb` packages that we build as it is now end-of-life. Contributed by @gary-kim.
changelog.d/7889.doc
+1
View File
@@ -0,0 +1 @@
Change the sample config postgres user section to use `synapse_user` instead of `synapse` to align with the documentation.
changelog.d/7890.misc
+1
View File
@@ -0,0 +1 @@
Fix typo in generated config file. Contributed by @ThiefMaster.
changelog.d/7892.misc
+1
View File
@@ -0,0 +1 @@
Import ABC from `collections.abc` for Python 3.10 compatibility.
changelog.d/7895.bugfix
+1
View File
@@ -0,0 +1 @@
Fix deprecation warning due to invalid escape sequences.
changelog.d/7897.misc
+2
View File
@@ -0,0 +1,2 @@
Remove unused functions `time_function`, `trace_function`, `get_previous_frames`
and `get_previous_frame` from `synapse.logging.utils` module.
changelog.d/7908.feature
+1
View File
@@ -0,0 +1 @@
Add the ability to re-activate an account from the admin API.
changelog.d/7912.misc
+1
View File
@@ -0,0 +1 @@
Convert `RoomListHandler` to async/await.
changelog.d/7914.misc
+1
View File
@@ -0,0 +1 @@
Lint the `contrib/` directory in CI and linting scripts, add `synctl` to the linting script for consistency with CI.
changelog.d/7919.misc
+1
View File
@@ -0,0 +1 @@
Use Element CSS and logo in notification emails when app name is Element.
changelog.d/7927.misc
+1
View File
@@ -0,0 +1 @@
Optimisation to /sync handling: skip serializing the response if the client has already disconnected.
changelog.d/7928.misc
+1
View File
@@ -0,0 +1 @@
When a client disconnects, don't log it as 'Error processing request'.
changelog.d/7929.misc
+1
View File
@@ -0,0 +1 @@
Add debugging to `/sync` response generation (disabled by default).
changelog.d/7930.feature
+1
View File
@@ -0,0 +1 @@
Abort federation requests where the client disconnects before the ratelimiter expires.
changelog.d/7931.feature
+1
View File
@@ -0,0 +1 @@
Cache responses to `/_matrix/federation/v1/state_ids` to reduce duplicated work.
changelog.d/7933.doc
+1
View File
@@ -0,0 +1 @@
Reorder database paragraphs to promote postgres over sqlite.
changelog.d/7934.doc
+1
View File
@@ -0,0 +1 @@
Update the dates of ACME v1's end of life in [`ACME.md`](https://github.com/matrix-org/synapse/blob/master/docs/ACME.md).
changelog.d/7935.misc
+1
View File
@@ -0,0 +1 @@
Convert the auth providers to be async/await.
changelog.d/7939.misc
+1
View File
@@ -0,0 +1 @@
Convert presence handler helpers to async/await.
contrib/cmdclient/console.py
+10 -11
View File
@@ -17,9 +17,6 @@
""" Starts a synapse client console. """
from __future__ import print_function
from twisted.internet import reactor, defer, threads
from http import TwistedHttpClient
import argparse
import cmd
import getpass
@@ -28,12 +25,14 @@ import shlex
import sys
import time
import urllib
import urlparse
from http import TwistedHttpClient
import nacl.signing
import nacl.encoding
import nacl.signing
import urlparse
from signedjson.sign import SignatureVerifyException, verify_signed_json
from signedjson.sign import verify_signed_json, SignatureVerifyException
from twisted.internet import defer, reactor, threads
CONFIG_JSON = "cmdclient_config.json"
@@ -493,7 +492,7 @@ class SynapseCmd(cmd.Cmd):
"list messages <roomid> from=END&to=START&limit=3"
"""
args = self._parse(line, ["type", "roomid", "qp"])
if not "type" in args or not "roomid" in args:
if "type" not in args or "roomid" not in args:
print("Must specify type and room ID.")
return
if args["type"] not in ["members", "messages"]:
@@ -508,7 +507,7 @@ class SynapseCmd(cmd.Cmd):
try:
key_value = key_value_str.split("=")
qp[key_value[0]] = key_value[1]
except:
except Exception:
print("Bad query param: %s" % key_value)
return
@@ -585,7 +584,7 @@ class SynapseCmd(cmd.Cmd):
parsed_url = urlparse.urlparse(args["path"])
qp.update(urlparse.parse_qs(parsed_url.query))
args["path"] = parsed_url.path
except:
except Exception:
pass
reactor.callFromThread(
@@ -772,10 +771,10 @@ def main(server_url, identity_server_url, username, token, config_path):
syn_cmd.config = json.load(config)
try:
http_client.verbose = "on" == syn_cmd.config["verbose"]
except:
except Exception:
pass
print("Loaded config from %s" % config_path)
except:
except Exception:
pass
# Twisted-specific: Runs the command processor in Twisted's event loop
contrib/cmdclient/http.py
+5 -5
View File
@@ -14,14 +14,14 @@
# limitations under the License.
from __future__ import print_function
from twisted.web.client import Agent, readBody
from twisted.web.http_headers import Headers
from twisted.internet import defer, reactor
from pprint import pformat
import json
import urllib
from pprint import pformat
from twisted.internet import defer, reactor
from twisted.web.client import Agent, readBody
from twisted.web.http_headers import Headers
class HttpClient(object):
contrib/experiments/test_messaging.py
+21 -34
View File
@@ -28,27 +28,24 @@ Currently assumes the local address is localhost:<port>
"""
from synapse.federation import ReplicationHandler
from synapse.federation.units import Pdu
from synapse.util import origin_from_ucid
from synapse.app.homeserver import SynapseHomeServer
# from synapse.logging.utils import log_function
from twisted.internet import reactor, defer
from twisted.python import log
import argparse
import curses.wrapper
import json
import logging
import os
import re
import cursesio
import curses.wrapper
from twisted.internet import defer, reactor
from twisted.python import log
from synapse.app.homeserver import SynapseHomeServer
from synapse.federation import ReplicationHandler
from synapse.federation.units import Pdu
from synapse.util import origin_from_ucid
# from synapse.logging.utils import log_function
logger = logging.getLogger("example")
@@ -75,7 +72,7 @@ class InputOutput(object):
"""
try:
m = re.match("^join (\S+)$", line)
m = re.match(r"^join (\S+)$", line)
if m:
# The `sender` wants to join a room.
(room_name,) = m.groups()
@@ -84,7 +81,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^invite (\S+) (\S+)$", line)
m = re.match(r"^invite (\S+) (\S+)$", line)
if m:
# `sender` wants to invite someone to a room
room_name, invitee = m.groups()
@@ -93,7 +90,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^send (\S+) (.*)$", line)
m = re.match(r"^send (\S+) (.*)$", line)
if m:
# `sender` wants to message a room
room_name, body = m.groups()
@@ -102,7 +99,7 @@ class InputOutput(object):
# self.print_line("OK.")
return
m = re.match("^backfill (\S+)$", line)
m = re.match(r"^backfill (\S+)$", line)
if m:
# we want to backfill a room
(room_name,) = m.groups()
@@ -201,16 +198,6 @@ class HomeServer(ReplicationHandler):
% (pdu.context, pdu.pdu_type, json.dumps(pdu.content))
)
# def on_state_change(self, pdu):
##self.output.print_line("#%s (state) %s *** %s" %
##(pdu.context, pdu.state_key, pdu.pdu_type)
##)
# if "joinee" in pdu.content:
# self._on_join(pdu.context, pdu.content["joinee"])
# elif "invitee" in pdu.content:
# self._on_invite(pdu.origin, pdu.context, pdu.content["invitee"])
def _on_message(self, pdu):
""" We received a message
"""
@@ -314,7 +301,7 @@ class HomeServer(ReplicationHandler):
return self.replication_layer.backfill(dest, room_name, limit)
def _get_room_remote_servers(self, room_name):
return [i for i in self.joined_rooms.setdefault(room_name).servers]
return list(self.joined_rooms.setdefault(room_name).servers)
def _get_or_create_room(self, room_name):
return self.joined_rooms.setdefault(room_name, Room(room_name))
@@ -334,7 +321,7 @@ def main(stdscr):
user = args.user
server_name = origin_from_ucid(user)
## Set up logging ##
# Set up logging
root_logger = logging.getLogger()
@@ -354,7 +341,7 @@ def main(stdscr):
observer = log.PythonLoggingObserver()
observer.start()
## Set up synapse server
# Set up synapse server
curses_stdio = cursesio.CursesStdIO(stdscr)
input_output = InputOutput(curses_stdio, user)
@@ -368,16 +355,16 @@ def main(stdscr):
input_output.set_home_server(hs)
## Add input_output logger
# Add input_output logger
io_logger = IOLoggerHandler(input_output)
io_logger.setFormatter(formatter)
root_logger.addHandler(io_logger)
## Start! ##
# Start!
try:
port = int(server_name.split(":")[1])
except:
except Exception:
port = 12345
app_hs.get_http_server().start_listening(port)
contrib/grafana/synapse.json
+214 -85
View File
@@ -1,7 +1,44 @@
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "6.7.4"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": ""
},
{
"type": "panel",
"id": "heatmap",
"name": "Heatmap",
"version": ""
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
}
],
"annotations": {
"list": [
{
"$$hashKey": "object:76",
"builtIn": 1,
"datasource": "$datasource",
"enable": false,
@@ -17,8 +54,8 @@
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 1,
"iteration": 1591098104645,
"id": null,
"iteration": 1594646317221,
"links": [
{
"asDropdown": true,
@@ -34,7 +71,7 @@
"panels": [
{
"collapsed": false,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -269,7 +306,6 @@
"show": false
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@@ -559,7 +595,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -1423,7 +1459,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -1795,7 +1831,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -2531,7 +2567,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -2823,7 +2859,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -2844,7 +2880,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 33
"y": 6
},
"hiddenSeries": false,
"id": 79,
@@ -2940,7 +2976,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 33
"y": 6
},
"hiddenSeries": false,
"id": 83,
@@ -3038,7 +3074,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 42
"y": 15
},
"hiddenSeries": false,
"id": 109,
@@ -3137,7 +3173,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 42
"y": 15
},
"hiddenSeries": false,
"id": 111,
@@ -3223,14 +3259,14 @@
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"description": "",
"description": "Number of events queued up on the master process for processing by the federation sender",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 51
"y": 24
},
"hiddenSeries": false,
"id": 140,
@@ -3354,6 +3390,103 @@
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "${DS_PROMETHEUS}",
"description": "The number of events in the in-memory queues ",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"hiddenSeries": false,
"id": 142,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "synapse_federation_transaction_queue_pending_pdus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"interval": "",
"legendFormat": "pending PDUs {{job}}-{{index}}",
"refId": "A"
},
{
"expr": "synapse_federation_transaction_queue_pending_edus{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"interval": "",
"legendFormat": "pending EDUs {{job}}-{{index}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "In-memory federation transmission queues",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"$$hashKey": "object:317",
"format": "short",
"label": "events",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"$$hashKey": "object:318",
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Federation",
@@ -3361,7 +3494,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -3567,7 +3700,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -3588,7 +3721,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 52
"y": 79
},
"hiddenSeries": false,
"id": 48,
@@ -3682,7 +3815,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 52
"y": 79
},
"hiddenSeries": false,
"id": 104,
@@ -3802,7 +3935,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 59
"y": 86
},
"hiddenSeries": false,
"id": 10,
@@ -3898,7 +4031,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 59
"y": 86
},
"hiddenSeries": false,
"id": 11,
@@ -3987,7 +4120,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -4011,7 +4144,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 67
"y": 80
},
"hiddenSeries": false,
"id": 12,
@@ -4106,7 +4239,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 67
"y": 80
},
"hiddenSeries": false,
"id": 26,
@@ -4201,7 +4334,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 80
"y": 93
},
"hiddenSeries": false,
"id": 13,
@@ -4297,7 +4430,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 80
"y": 93
},
"hiddenSeries": false,
"id": 27,
@@ -4392,7 +4525,7 @@
"h": 13,
"w": 12,
"x": 0,
"y": 93
"y": 106
},
"hiddenSeries": false,
"id": 28,
@@ -4486,7 +4619,7 @@
"h": 13,
"w": 12,
"x": 12,
"y": 93
"y": 106
},
"hiddenSeries": false,
"id": 25,
@@ -4572,7 +4705,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -5062,7 +5195,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -5083,7 +5216,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 66
"y": 121
},
"hiddenSeries": false,
"id": 91,
@@ -5179,7 +5312,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 66
"y": 121
},
"hiddenSeries": false,
"id": 21,
@@ -5271,7 +5404,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 75
"y": 130
},
"hiddenSeries": false,
"id": 89,
@@ -5369,7 +5502,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 75
"y": 130
},
"hiddenSeries": false,
"id": 93,
@@ -5459,7 +5592,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 84
"y": 139
},
"hiddenSeries": false,
"id": 95,
@@ -5552,12 +5685,12 @@
"mode": "spectrum"
},
"dataFormat": "tsbuckets",
"datasource": "Prometheus",
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 84
"y": 139
},
"heatmap": {},
"hideZeroBuckets": true,
@@ -5567,7 +5700,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@@ -5609,7 +5741,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -5630,7 +5762,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 39
"y": 66
},
"hiddenSeries": false,
"id": 2,
@@ -5754,7 +5886,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 39
"y": 66
},
"hiddenSeries": false,
"id": 41,
@@ -5847,7 +5979,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 46
"y": 73
},
"hiddenSeries": false,
"id": 42,
@@ -5939,7 +6071,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 46
"y": 73
},
"hiddenSeries": false,
"id": 43,
@@ -6031,7 +6163,7 @@
"h": 7,
"w": 12,
"x": 0,
"y": 53
"y": 80
},
"hiddenSeries": false,
"id": 113,
@@ -6129,7 +6261,7 @@
"h": 7,
"w": 12,
"x": 12,
"y": 53
"y": 80
},
"hiddenSeries": false,
"id": 115,
@@ -6215,7 +6347,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -6236,7 +6368,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 58
"y": 40
},
"hiddenSeries": false,
"id": 67,
@@ -6267,7 +6399,7 @@
"steppedLine": false,
"targets": [
{
"expr": " synapse_event_persisted_position{instance=\"$instance\",job=\"synapse\"} - ignoring(index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"expr": "max(synapse_event_persisted_position{instance=\"$instance\"}) - ignoring(instance,index, job, name) group_right() synapse_event_processing_positions{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
@@ -6328,7 +6460,7 @@
"h": 9,
"w": 12,
"x": 12,
"y": 58
"y": 40
},
"hiddenSeries": false,
"id": 71,
@@ -6362,6 +6494,7 @@
"expr": "time()*1000-synapse_event_processing_last_ts{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{job}}-{{index}} {{name}}",
"refId": "B"
@@ -6420,7 +6553,7 @@
"h": 9,
"w": 12,
"x": 0,
"y": 67
"y": 49
},
"hiddenSeries": false,
"id": 121,
@@ -6509,7 +6642,7 @@
},
{
"collapsed": true,
"datasource": null,
"datasource": "${DS_PROMETHEUS}",
"gridPos": {
"h": 1,
"w": 24,
@@ -6539,7 +6672,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 41
"y": 86
},
"heatmap": {},
"hideZeroBuckets": true,
@@ -6549,7 +6682,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@@ -6599,7 +6731,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 41
"y": 86
},
"hiddenSeries": false,
"id": 124,
@@ -6700,7 +6832,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 49
"y": 94
},
"heatmap": {},
"hideZeroBuckets": true,
@@ -6710,7 +6842,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@@ -6760,7 +6891,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 49
"y": 94
},
"hiddenSeries": false,
"id": 128,
@@ -6879,7 +7010,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 57
"y": 102
},
"heatmap": {},
"hideZeroBuckets": true,
@@ -6889,7 +7020,6 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
@@ -6939,7 +7069,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 57
"y": 102
},
"hiddenSeries": false,
"id": 130,
@@ -7058,7 +7188,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 65
"y": 110
},
"heatmap": {},
"hideZeroBuckets": true,
@@ -7068,12 +7198,12 @@
"show": true
},
"links": [],
"options": {},
"reverseYBuckets": false,
"targets": [
{
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)",
"expr": "rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size])",
"format": "heatmap",
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{le}}",
"refId": "A"
@@ -7118,7 +7248,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 65
"y": 110
},
"hiddenSeries": false,
"id": 132,
@@ -7149,29 +7279,33 @@
"steppedLine": false,
"targets": [
{
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0)) ",
"expr": "histogram_quantile(0.5, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "50%",
"refId": "A"
},
{
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.75, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "75%",
"refId": "B"
},
{
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.90, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "90%",
"refId": "C"
},
{
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\"}[$bucket_size]) and on (index, instance, job) (synapse_storage_events_persisted_events > 0))",
"expr": "histogram_quantile(0.99, rate(synapse_state_number_state_groups_in_resolution_bucket{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size]))",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "99%",
"refId": "D"
@@ -7181,7 +7315,7 @@
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Number of state resolution performed, by number of state groups involved (quantiles)",
"title": "Number of state resolutions performed, by number of state groups involved (quantiles)",
"tooltip": {
"shared": true,
"sort": 0,
@@ -7233,6 +7367,7 @@
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "Prometheus"
},
@@ -7309,14 +7444,12 @@
},
{
"allValue": null,
"current": {
"text": "matrix.org",
"value": "matrix.org"
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"includeAll": false,
"index": -1,
"label": null,
"multi": false,
"name": "instance",
@@ -7335,17 +7468,13 @@
{
"allFormat": "regex wildcard",
"allValue": "",
"current": {
"text": "synapse",
"value": [
"synapse"
]
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"hideLabel": false,
"includeAll": true,
"index": -1,
"label": "Job",
"multi": true,
"multiFormat": "regex values",
@@ -7366,16 +7495,13 @@
{
"allFormat": "regex wildcard",
"allValue": ".*",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"current": {},
"datasource": "$datasource",
"definition": "",
"hide": 0,
"hideLabel": false,
"includeAll": true,
"index": -1,
"label": "",
"multi": true,
"multiFormat": "regex values",
@@ -7428,5 +7554,8 @@
"timezone": "",
"title": "Synapse",
"uid": "000000012",
"version": 29
"variables": {
"list": []
},
"version": 32
}
contrib/graph/graph.py
+10 -11
View File
@@ -1,5 +1,13 @@
from __future__ import print_function
import argparse
import cgi
import datetime
import json
import pydot
import urllib2
# Copyright 2014-2016 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,15 +23,6 @@ from __future__ import print_function
# limitations under the License.
import sqlite3
import pydot
import cgi
import json
import datetime
import argparse
import urllib2
def make_name(pdu_id, origin):
return "%s@%s" % (pdu_id, origin)
@@ -33,7 +32,7 @@ def make_graph(pdus, room, filename_prefix):
node_map = {}
origins = set()
colors = set(("red", "green", "blue", "yellow", "purple"))
colors = {"red", "green", "blue", "yellow", "purple"}
for pdu in pdus:
origins.add(pdu.get("origin"))
@@ -49,7 +48,7 @@ def make_graph(pdus, room, filename_prefix):
try:
c = colors.pop()
color_map[o] = c
except:
except Exception:
print("Run out of colours!")
color_map[o] = "black"
contrib/graph/graph2.py
+7 -6
View File
@@ -13,12 +13,13 @@
# limitations under the License.
import sqlite3
import pydot
import cgi
import json
import datetime
import argparse
import cgi
import datetime
import json
import sqlite3
import pydot
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
@@ -98,7 +99,7 @@ def make_graph(db_name, room_id, file_prefix, limit):
for prev_id, _ in event.prev_events:
try:
end_node = node_map[prev_id]
except:
except Exception:
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
node_map[prev_id] = end_node
contrib/graph/graph3.py
+11 -11
View File
@@ -1,5 +1,15 @@
from __future__ import print_function
import argparse
import cgi
import datetime
import pydot
import simplejson as json
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
# Copyright 2016 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,16 +25,6 @@ from __future__ import print_function
# limitations under the License.
import pydot
import cgi
import simplejson as json
import datetime
import argparse
from synapse.events import FrozenEvent
from synapse.util.frozenutils import unfreeze
def make_graph(file_name, room_id, file_prefix, limit):
print("Reading lines")
with open(file_name) as f:
@@ -106,7 +106,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
for prev_id, _ in event.prev_events:
try:
end_node = node_map[prev_id]
except:
except Exception:
end_node = pydot.Node(name=prev_id, label="<<b>%s</b>>" % (prev_id,))
node_map[prev_id] = end_node
contrib/jitsimeetbridge/jitsimeetbridge.py
+5 -5
View File
@@ -12,15 +12,15 @@ npm install jquery jsdom
"""
from __future__ import print_function
import gevent
import grequests
from BeautifulSoup import BeautifulSoup
import json
import urllib
import subprocess
import time
# ACCESS_TOKEN="" #
import gevent
import grequests
from BeautifulSoup import BeautifulSoup
ACCESS_TOKEN = ""
MATRIXBASE = "https://matrix.org/_matrix/client/api/v1/"
MYUSERNAME = "@davetest:matrix.org"
contrib/scripts/kick_users.py
+4 -2
View File
@@ -1,10 +1,12 @@
#!/usr/bin/env python
from __future__ import print_function
from argparse import ArgumentParser
import json
import requests
import sys
import urllib
from argparse import ArgumentParser
import requests
try:
raw_input
debian/changelog
Vendored
+18
View File
@@ -1,3 +1,21 @@
matrix-synapse-py3 (1.17.0) stable; urgency=medium
* New synapse release 1.17.0.
-- Synapse Packaging team <packages@matrix.org> Mon, 13 Jul 2020 10:20:31 +0100
matrix-synapse-py3 (1.16.1) stable; urgency=medium
* New synapse release 1.16.1.
-- Synapse Packaging team <packages@matrix.org> Fri, 10 Jul 2020 12:09:24 +0100
matrix-synapse-py3 (1.17.0rc1) stable; urgency=medium
* New synapse release 1.17.0rc1.
-- Synapse Packaging team <packages@matrix.org> Thu, 09 Jul 2020 16:53:12 +0100
matrix-synapse-py3 (1.16.0) stable; urgency=medium
* New synapse release 1.16.0.

Some files were not shown because too many files have changed in this diff Show More