a1e9abc7df
Add Prometheus [HTTP service discovery](https://prometheus.io/docs/prometheus/latest/http_sd/) endpoint for easy discovery of all workers in Docker image. Follow-up to https://github.com/element-hq/synapse/pull/19324 Spawning from wanting to [run a load test](https://github.com/element-hq/synapse-rust-apps/pull/397) against the Complement Docker image of Synapse and see metrics from the homeserver. `GET http://<synapse_container>:9469/metrics/service_discovery` ```json5 [ { "targets": [ "<host>", ... ], "labels": { "<labelname>": "<labelvalue>", ... } }, ... ] ``` The metrics from each worker can also be accessed via `http://<synapse_container>:9469/metrics/worker/<worker_name>` which is what the service discovery response points to behind the scenes. This way, you only need to expose a single port (9469) to access all metrics. <details> <summary>Real HTTP service discovery response</summary> ```json5 [ { "targets": [ "localhost:9469" ], "labels": { "job": "event_persister", "index": "1", "__metrics_path__": "/metrics/worker/event_persister1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "event_persister", "index": "2", "__metrics_path__": "/metrics/worker/event_persister2" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "background_worker", "index": "1", "__metrics_path__": "/metrics/worker/background_worker1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "event_creator", "index": "1", "__metrics_path__": "/metrics/worker/event_creator1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "user_dir", "index": "1", "__metrics_path__": "/metrics/worker/user_dir1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "media_repository", "index": "1", "__metrics_path__": "/metrics/worker/media_repository1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "federation_inbound", "index": "1", "__metrics_path__": "/metrics/worker/federation_inbound1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "federation_reader", "index": "1", "__metrics_path__": "/metrics/worker/federation_reader1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "federation_sender", "index": "1", "__metrics_path__": "/metrics/worker/federation_sender1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "synchrotron", "index": "1", "__metrics_path__": "/metrics/worker/synchrotron1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "client_reader", "index": "1", "__metrics_path__": "/metrics/worker/client_reader1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "appservice", "index": "1", "__metrics_path__": "/metrics/worker/appservice1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "pusher", "index": "1", "__metrics_path__": "/metrics/worker/pusher1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "device_lists", "index": "1", "__metrics_path__": "/metrics/worker/device_lists1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "device_lists", "index": "2", "__metrics_path__": "/metrics/worker/device_lists2" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "stream_writers", "index": "1", "__metrics_path__": "/metrics/worker/stream_writers1" } }, { "targets": [ "localhost:9469" ], "labels": { "job": "main", "index": "1", "__metrics_path__": "/metrics/worker/main" } } ] ``` </details> And how it ends up as targets in Prometheus (http://localhost:9090/targets): (image) ### Testing strategy 1. Make sure your firewall allows the Docker containers to communicate to the host (`host.docker.internal`) so they can access exposed ports of other Docker containers. We want to allow Synapse to access the Prometheus container and Grafana to access to the Prometheus container. - `sudo ufw allow in on docker0 comment "Allow traffic from the default Docker network to the host machine (host.docker.internal)"` - `sudo ufw allow in on br-+ comment "(from Matrix Complement testing) Allow traffic from custom Docker networks to the host machine (host.docker.internal)"` - [Complement firewall docs](ee6acd9154/README.md (potential-conflict-with-firewall-software)) 1. Build the Docker image for Synapse: `docker build -t matrixdotorg/synapse -f docker/Dockerfile . && docker build -t matrixdotorg/synapse-workers -f docker/Dockerfile-workers .` ([docs](7a24fafbc3/docker/README-testing.md (building-and-running-the-images-manually))) 1. Start Synapse: ``` docker run -d --name synapse \ --mount type=volume,src=synapse-data,dst=/data \ -e SYNAPSE_SERVER_NAME=my.docker.synapse.server \ -e SYNAPSE_REPORT_STATS=no \ -e SYNAPSE_ENABLE_METRICS=1 \ -p 8008:8008 \ -p 9469:9469 \ matrixdotorg/synapse-workers:latest ``` - Also try with workers: ``` docker run -d --name synapse \ --mount type=volume,src=synapse-data,dst=/data \ -e SYNAPSE_SERVER_NAME=my.docker.synapse.server \ -e SYNAPSE_REPORT_STATS=no \ -e SYNAPSE_ENABLE_METRICS=1 \ -e SYNAPSE_WORKER_TYPES="\ event_persister:2, \ background_worker, \ event_creator, \ user_dir, \ media_repository, \ federation_inbound, \ federation_reader, \ federation_sender, \ synchrotron, \ client_reader, \ appservice, \ pusher, \ device_lists:2, \ stream_writers=account_data+presence+receipts+to_device+typing" \ -p 8008:8008 \ -p 9469:9469 \ matrixdotorg/synapse-workers:latest ``` 1. You should be able to see Prometheus service discovery endpoint at http://localhost:9469/metrics/service_discovery 1. Create a Prometheus config (`prometheus.yml`) ```yaml global: scrape_interval: 15s scrape_timeout: 15s evaluation_interval: 15s scrape_configs: - job_name: synapse scrape_interval: 15s metrics_path: /_synapse/metrics scheme: http # We set `honor_labels` so that each service can set their own `job` label # # > honor_labels controls how Prometheus handles conflicts between labels that are # > already present in scraped data and labels that Prometheus would attach # > server-side ("job" and "instance" labels, manually configured target # > labels, and labels generated by service discovery implementations). # > # > *-- https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config* honor_labels: true # Use HTTP service discovery # # Reference: # - https://prometheus.io/docs/prometheus/latest/http_sd/ # - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config http_sd_configs: - url: 'http://localhost:9469/metrics/service_discovery' ``` 1. Start Prometheus (update the volume bind mount to the config you just saved somewhere): ``` docker run \ --detach \ --name=prometheus \ --add-host host.docker.internal:host-gateway \ -p 9090:9090 \ -v ~/Documents/code/random/prometheus-config/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus ``` 1. Make sure you're seeing some data in Prometheus. On http://localhost:9090/query, search for `synapse_build_info` 1. Start [Grafana](https://hub.docker.com/r/grafana/grafana) ``` docker run -d --name=grafana --add-host host.docker.internal:host-gateway -p 3000:3000 grafana/grafana ``` 1. Visit the Grafana dashboard, http://localhost:3000/ (Credentials: `admin`/`admin`) 1. **Connections** -> **Data Sources** -> **Add data source** -> **Prometheus** - Prometheus server URL: `http://host.docker.internal:9090` 1. Import the Synapse dashboard: https://github.com/element-hq/synapse/blob/develop/contrib/grafana/synapse.json
.. image:: https://github.com/element-hq/synapse/raw/develop/docs/element_logo_white_bg.svg :height: 60px **Element Synapse - Matrix homeserver implementation** |support| |development| |documentation| |license| |pypi| |python| Synapse is an open source `Matrix <https://matrix.org>`__ homeserver implementation, written and maintained by `Element <https://element.io>`_. `Matrix <https://github.com/matrix-org>`__ is the open standard for secure and interoperable real-time communications. You can directly run and manage the source code in this repository, available under an AGPL license (or alternatively under a commercial license from Element). There is no support provided by Element unless you have a subscription from Element. 🚀 Getting started ================== This component is developed and maintained by `Element <https://element.io>`_. It gets shipped as part of the **Element Server Suite (ESS)** which provides the official means of deployment. ESS is a Matrix distribution from Element with focus on quality and ease of use. It ships a full Matrix stack tailored to the respective use case. There are three editions of ESS: - `ESS Community <https://github.com/element-hq/ess-helm>`_ - the free Matrix distribution from Element tailored to small-/mid-scale, non-commercial community use cases - `ESS Pro <https://element.io/server-suite>`_ - the commercial Matrix distribution from Element for professional use - `ESS TI-M <https://element.io/server-suite/ti-messenger>`_ - a special version of ESS Pro focused on the requirements of TI-Messenger Pro and ePA as specified by the German National Digital Health Agency Gematik 🛠️ Standalone installation and configuration ============================================ The Synapse documentation describes `options for installing Synapse standalone <https://element-hq.github.io/synapse/latest/setup/installation.html>`_. See below for more useful documentation links. - `Synapse configuration options <https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html>`_ - `Synapse configuration for federation <https://element-hq.github.io/synapse/latest/federate.html>`_ - `Using a reverse proxy with Synapse <https://element-hq.github.io/synapse/latest/reverse_proxy.html>`_ - `Upgrading Synapse <https://element-hq.github.io/synapse/develop/upgrade.html>`_ 🎯 Troubleshooting and support ============================== 🚀 Professional support ----------------------- Enterprise quality support for Synapse including SLAs is available as part of an `Element Server Suite (ESS) <https://element.io/pricing>`_ subscription. If you are an existing ESS subscriber then you can raise a `support request <https://ems.element.io/support>`_ and access the `Element product documentation <https://docs.element.io>`_. 🤝 Community support -------------------- The `Admin FAQ <https://element-hq.github.io/synapse/latest/usage/administration/admin_faq.html>`_ includes tips on dealing with some common problems. For more details, see `Synapse's wider documentation <https://element-hq.github.io/synapse/latest/>`_. For additional support installing or managing Synapse, please ask in the community support room |room|_ (from a matrix.org account if necessary). We do not use GitHub issues for support requests, only for bug reports and feature requests. .. |room| replace:: ``#synapse:matrix.org`` .. _room: https://matrix.to/#/#synapse:matrix.org .. |docs| replace:: ``docs`` .. _docs: docs 🛠️ Development ============== We welcome contributions to Synapse from the community! The best place to get started is our `guide for contributors <https://element-hq.github.io/synapse/latest/development/contributing_guide.html>`_. This is part of our broader `documentation <https://element-hq.github.io/synapse/latest>`_, which includes information for Synapse developers as well as Synapse administrators. Developers might be particularly interested in: * `Synapse's database schema <https://element-hq.github.io/synapse/latest/development/database_schema.html>`_, * `notes on Synapse's implementation details <https://element-hq.github.io/synapse/latest/development/internal_documentation/index.html>`_, and * `how we use git <https://element-hq.github.io/synapse/latest/development/git.html>`_. Alongside all that, join our developer community on Matrix: `#synapse-dev:matrix.org <https://matrix.to/#/#synapse-dev:matrix.org>`_, featuring real humans! Copyright and Licensing ======================= | Copyright 2014–2017 OpenMarket Ltd | Copyright 2017 Vector Creations Ltd | Copyright 2017–2025 New Vector Ltd | Copyright 2025 Element Creations Ltd This software is dual-licensed by Element Creations Ltd (Element). It can be used either: (1) for free under the terms of the GNU Affero General Public License (as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version); OR (2) under the terms of a paid-for Element Commercial License agreement between you and Element (the terms of which may vary depending on what you and Element have agreed to). Unless required by applicable law or agreed to in writing, software distributed under the Licenses is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the Licenses for the specific language governing permissions and limitations under the Licenses. Please contact `licensing@element.io <mailto:licensing@element.io>`_ to purchase an Element commercial license for this software. .. |support| image:: https://img.shields.io/badge/matrix-community%20support-success :alt: (get community support in #synapse:matrix.org) :target: https://matrix.to/#/#synapse:matrix.org .. |development| image:: https://img.shields.io/matrix/synapse-dev:matrix.org?label=development&logo=matrix :alt: (discuss development on #synapse-dev:matrix.org) :target: https://matrix.to/#/#synapse-dev:matrix.org .. |documentation| image:: https://img.shields.io/badge/documentation-%E2%9C%93-success :alt: (Rendered documentation on GitHub Pages) :target: https://element-hq.github.io/synapse/latest/ .. |license| image:: https://img.shields.io/github/license/element-hq/synapse :alt: (check license in LICENSE file) :target: LICENSE .. |pypi| image:: https://img.shields.io/pypi/v/matrix-synapse :alt: (latest version released on PyPi) :target: https://pypi.org/project/matrix-synapse .. |python| image:: https://img.shields.io/pypi/pyversions/matrix-synapse :alt: (supported python versions) :target: https://pypi.org/project/matrix-synapse