58f59ffbcb
- Update `synapse_xxx` (server-level) metrics to use
`server_name="$server_name",` instead of `instance="$instance"`
- Add `synapse_server_name_info` metric to map Synapse `server_name`s to
the `instance`s they're hosted on.
- For process level metrics, update to use `xxx * on (instance, job,
index) group_left(server_name)
synapse_server_name_info{server_name="$server_name"}`
All of the changes here are backwards compatible with whatever people
were doing before with their Prometheus/Grafana dashboards.
Previously, the recommendation was to use the `instance` label to group
everything under the same server (803e4b4d88/docs/metrics-howto.md (L93-L147))
But the `instance` label actually has a special meaning and we're
actually abusing it by using it that way:
> `instance`: The `<host>:<port>` part of the target's URL that was
scraped.
>
> *--
https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series*
Since https://github.com/element-hq/synapse/issues/18592 (Synapse
`v1.139.0`), we now have the `server_name` label to use instead.
---
Additionally, the assumption that a single process is serving a single
server is no longer true with [Synapse Pro for small
hosts](https://docs.element.io/latest/element-server-suite-pro/synapse-pro-for-small-hosts/overview/).
Part of https://github.com/element-hq/synapse-small-hosts/issues/106
### Motivating use case
Although this change also benefits [Synapse Pro for small
hosts](https://docs.element.io/latest/element-server-suite-pro/synapse-pro-for-small-hosts/overview/)
(https://github.com/element-hq/synapse-small-hosts/issues/106), this is
actually spawning from adding Prometheus metrics to our workerized
Docker image (https://github.com/element-hq/synapse/pull/19324,
https://github.com/element-hq/synapse/pull/19336) with a more correct
label setup (without `instance`) and wanting the dashboard to be better.
### Testing strategy
1. Make sure your firewall allows the Docker containers to communicate
to the host (`host.docker.internal`) so they can access exposed ports of
other Docker containers. We want to allow Synapse to access the
Prometheus container and Grafana to access to the Prometheus container.
- `sudo ufw allow in on docker0 comment "Allow traffic from the default
Docker network to the host machine (host.docker.internal)"`
- `sudo ufw allow in on br-+ comment "(from Matrix Complement testing)
Allow traffic from custom Docker networks to the host machine
(host.docker.internal)"`
- [Complement firewall
docs](ee6acd9154/README.md (potential-conflict-with-firewall-software))
1. Build the Docker image for Synapse: `docker build -t
matrixdotorg/synapse -f docker/Dockerfile .`
([docs](7a24fafbc3/docker/README-testing.md (building-and-running-the-images-manually)))
1. Generate config for Synapse:
```
docker run -it --rm \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
-e SYNAPSE_REPORT_STATS=yes \
-e SYNAPSE_ENABLE_METRICS=1 \
matrixdotorg/synapse:latest generate
```
1. Start Synapse:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-p 8008:8008 \
-p 19090:19090 \
matrixdotorg/synapse:latest
```
1. You should be able to see metrics from Synapse at
http://localhost:19090/_synapse/metrics
1. Create a Prometheus config (`prometheus.yml`)
```yaml
global:
scrape_interval: 15s
scrape_timeout: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: prometheus
scrape_interval: 15s
metrics_path: /_synapse/metrics
scheme: http
static_configs:
- targets:
# This should point to the Synapse metrics listener (we're using
`host.docker.internal` because this is from within the Prometheus
container)
- host.docker.internal:19090
```
1. Start Prometheus (update the volume bind mount to the config you just
saved somewhere):
```
docker run \
--detach \
--name=prometheus \
--add-host host.docker.internal:host-gateway \
-p 9090:9090 \
-v
~/Documents/code/random/prometheus-config/prometheus.yml:/etc/prometheus/prometheus.yml
\
prom/prometheus
```
1. Make sure you're seeing some data in Prometheus. On
http://localhost:9090/query, search for `synapse_build_info`
1. Start [Grafana](https://hub.docker.com/r/grafana/grafana)
```
docker run -d --name=grafana --add-host
host.docker.internal:host-gateway -p 3000:3000 grafana/grafana
```
1. Visit the Grafana dashboard, http://localhost:3000/ (Credentials:
`admin`/`admin`)
1. **Connections** -> **Data Sources** -> **Add data source** ->
**Prometheus**
- Prometheus server URL: `http://host.docker.internal:9090`
1. Import the Synapse dashboard: `contrib/grafana/synapse.json`
To test workers, you can use the testing strategy from
https://github.com/element-hq/synapse/pull/19336 (assumes both changes
from this PR and the other PR are combined)
Synapse Documentation
The documentation is currently hosted here. Please update any links to point to the new website instead.
About
This directory currently holds a series of markdown files documenting how to install, use and develop Synapse. The documentation is readable directly from this repository, but it is recommended to instead browse through the website for easier discoverability.
Adding to the documentation
Most of the documentation currently exists as top-level files, as when organising them into
a structured website, these files were kept in place so that existing links would not break.
The rest of the documentation is stored in folders, such as setup, usage, and development
etc. All new documentation files should be placed in structured folders. For example:
To create a new user-facing documentation page about a new Single Sign-On protocol named "MyCoolProtocol", one should create a new file with a relevant name, such as "my_cool_protocol.md". This file might fit into the documentation structure at:
- Usage
- Configuration
- User Authentication
- Single Sign-On
- My Cool Protocol
- Single Sign-On
- User Authentication
- Configuration
Given that, one would place the new file under
usage/configuration/user_authentication/single_sign_on/my_cool_protocol.md.
Note that the structure of the documentation (and thus the left sidebar on the website) is determined by the list in SUMMARY.md. The final thing to do when adding a new page is to add a new line linking to the new documentation file:
- [My Cool Protocol](usage/configuration/user_authentication/single_sign_on/my_cool_protocol.md)
Building the documentation
The documentation is built with mdbook, and the outline of the documentation is determined by the structure of SUMMARY.md.
First, get mdbook. Then, from the root of the repository, build the documentation with:
mdbook build
The rendered contents will be outputted to a new book/ directory at the root of the repository. Please note that
index.html is not built by default, it is created by copying over the file welcome_and_overview.html to index.html
during deployment. Thus, when running mdbook serve locally the book will initially show a 404 in place of the index
due to the above. Do not be alarmed!
You can also have mdbook host the docs on a local webserver with hot-reload functionality via:
mdbook serve
The URL at which the docs can be viewed at will be logged.
Synapse configuration documentation
The Configuration Manual page is generated from a YAML file, schema/synapse-config.schema.yaml. To add new options or modify existing ones, first edit that file, then run scripts-dev/gen_config_documentation.py to generate an updated Configuration Manual markdown file.
Build the book as described above to preview it in a web browser.
Configuration and theming
The look and behaviour of the website is configured by the book.toml file at the root of the repository. See mdbook's documentation on configuration for available options.
The site can be themed and additionally extended with extra UI and features. See website_files/README.md for details.