From 6363d77ba299bec437911caf241ca34205cea172 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Thu, 15 Jan 2026 15:49:10 -0600 Subject: [PATCH] Warn about skipping reactor metrics when using unknown reactor type (#19383) Spawning from not seeing any reactor metrics in the Grafana dashboard in some load tests, noticing `python_twisted_reactor_tick_time_bucket` is `0` in Prometheus, following it back to Synapse and seeing that we don't warn about skipping reactor metrics in all cases (when using an unknown reactor type). A follow-up to this would be to actually figure out how to instrument the `ProxiedReactor` or why `ProxiedReactor` is being chosen in the first place and see if we can get it to use a more normal type :thinking: ### Reproduction instructions 1. Using the Complement scripts **with workers**: `WORKERS=1 ./scripts-dev/complement.sh ./tests/csapi` 1. `docker logs complement_csapi_dirty_hs1 2>&1 | grep -i "reactor"` 1. With these changes, notice `Skipping configuring ReactorLastSeenMetric: unexpected reactor type: <__main__.ProxiedReactor object at 0x7fc0adaaea50>` and `Twisted reactor: ProxiedReactor` 1. Cleanup: - `docker stop $(docker ps --all --filter "label=complement_context" --quiet)` - `docker rm $(docker ps --all --filter "label=complement_context" --quiet)` I'm unable to reproduce with the normal Synapse images or `complement-synapse` without workers. They all use `Twisted reactor: EPollReactor`
Checking docker/Dockerfile-workers 1. Build the Docker image for Synapse: `docker build -t matrixdotorg/synapse -f docker/Dockerfile . && docker build -t matrixdotorg/synapse-workers -f docker/Dockerfile-workers .` ([docs](https://github.com/element-hq/synapse/blob/7a24fafbc376b9bffeb3277b1ad4aa950720c96c/docker/README-testing.md#building-and-running-the-images-manually)) 1. Start Synapse: ``` docker run -d --name synapse \ --mount type=volume,src=synapse-data,dst=/data \ -e SYNAPSE_SERVER_NAME=my.docker.synapse.server \ -e SYNAPSE_REPORT_STATS=no \ -e SYNAPSE_ENABLE_METRICS=1 \ -p 8008:8008 \ -p 9469:9469 \ matrixdotorg/synapse-workers:latest ``` 1. `docker logs synapse 2>&1 | grep -i "reactor"` 1. Says `Twisted reactor: EPollReactor`
--- changelog.d/19383.misc | 1 + synapse/metrics/_reactor_metrics.py | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) create mode 100644 changelog.d/19383.misc diff --git a/changelog.d/19383.misc b/changelog.d/19383.misc new file mode 100644 index 0000000000..4749b1af9a --- /dev/null +++ b/changelog.d/19383.misc @@ -0,0 +1 @@ +Warn about skipping reactor metrics when using unknown reactor type. diff --git a/synapse/metrics/_reactor_metrics.py b/synapse/metrics/_reactor_metrics.py index 9852d0b932..e73ade9890 100644 --- a/synapse/metrics/_reactor_metrics.py +++ b/synapse/metrics/_reactor_metrics.py @@ -158,12 +158,17 @@ try: else: # E.g. this does not support the (Windows-only) ProactorEventLoop. logger.warning( - "Skipping configuring ReactorLastSeenMetric: unexpected asyncio loop selector: %r via %r", + "Skipping configuring reactor metrics: unexpected asyncio loop selector: %r via %r", selector, asyncio_loop, ) + else: + logger.warning( + "Skipping configuring reactor metrics: unexpected reactor type: %r", + reactor, + ) except Exception as e: - logger.warning("Configuring ReactorLastSeenMetric failed: %r", e) + logger.warning("Configuring reactor metrics failed: %r", e) if wrapper: