Warn about skipping reactor metrics when using unknown reactor type (#19383)
Spawning from not seeing any reactor metrics in the Grafana dashboard in
some load tests, noticing `python_twisted_reactor_tick_time_bucket` is
`0` in Prometheus, following it back to Synapse and seeing that we don't
warn about skipping reactor metrics in all cases (when using an unknown
reactor type).
A follow-up to this would be to actually figure out how to instrument
the `ProxiedReactor` or why `ProxiedReactor` is being chosen in the
first place and see if we can get it to use a more normal type
🤔
### Reproduction instructions
1. Using the Complement scripts **with workers**: `WORKERS=1
./scripts-dev/complement.sh ./tests/csapi`
1. `docker logs complement_csapi_dirty_hs1 2>&1 | grep -i "reactor"`
1. With these changes, notice `Skipping configuring
ReactorLastSeenMetric: unexpected reactor type: <__main__.ProxiedReactor
object at 0x7fc0adaaea50>` and `Twisted reactor: ProxiedReactor`
1. Cleanup:
- `docker stop $(docker ps --all --filter "label=complement_context"
--quiet)`
- `docker rm $(docker ps --all --filter "label=complement_context"
--quiet)`
I'm unable to reproduce with the normal Synapse images or
`complement-synapse` without workers. They all use `Twisted reactor:
EPollReactor`
<details>
<summary>Checking <code>docker/Dockerfile-workers</code></summary>
1. Build the Docker image for Synapse: `docker build -t
matrixdotorg/synapse -f docker/Dockerfile . && docker build -t
matrixdotorg/synapse-workers -f docker/Dockerfile-workers .`
([docs](7a24fafbc3/docker/README-testing.md (building-and-running-the-images-manually)))
1. Start Synapse:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
-e SYNAPSE_REPORT_STATS=no \
-e SYNAPSE_ENABLE_METRICS=1 \
-p 8008:8008 \
-p 9469:9469 \
matrixdotorg/synapse-workers:latest
```
1. `docker logs synapse 2>&1 | grep -i "reactor"`
1. Says `Twisted reactor: EPollReactor`
</details>
This commit is contained in:
1
changelog.d/19383.misc
Normal file
1
changelog.d/19383.misc
Normal file
@@ -0,0 +1 @@
|
||||
Warn about skipping reactor metrics when using unknown reactor type.
|
||||
@@ -158,12 +158,17 @@ try:
|
||||
else:
|
||||
# E.g. this does not support the (Windows-only) ProactorEventLoop.
|
||||
logger.warning(
|
||||
"Skipping configuring ReactorLastSeenMetric: unexpected asyncio loop selector: %r via %r",
|
||||
"Skipping configuring reactor metrics: unexpected asyncio loop selector: %r via %r",
|
||||
selector,
|
||||
asyncio_loop,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"Skipping configuring reactor metrics: unexpected reactor type: %r",
|
||||
reactor,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("Configuring ReactorLastSeenMetric failed: %r", e)
|
||||
logger.warning("Configuring reactor metrics failed: %r", e)
|
||||
|
||||
|
||||
if wrapper:
|
||||
|
||||
Reference in New Issue
Block a user