46d235cd52
This is useful so we can test Synapse specific behaviors like our admin API. (see docs in PR, `complement/README.md`) ``` COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo ``` Complement calls these ["out-of-repo"](78c255edce/OUT-OF-REPO-TESTS.md) tests but it's a bit of a misnomer once they're in your project. (just depends on the perspective) There has been [previous desire](https://github.com/element-hq/synapse/pull/19021#discussion_r2453442191) for this kind of thing but this is spawning from wanting to have some tests for our purge history admin API (https://github.com/element-hq/synapse-rust-apps/issues/430). There are some Sytest tests ([`matrix-org/sytest` -> `tests/48admin.pl#L91-L618`](1be04cce46/tests/48admin.pl (L91-L618))) for this already but I'd much rather work in Complement instead of Sytest. I'm wanting these tests to ensure that our new `event-cache` rust app for Synapse Pro doesn't break these kind of erasure features (https://github.com/element-hq/synapse-rust-apps/issues/366 and https://github.com/element-hq/synapse-rust-apps/issues/153). Interestingly, there is already [`matrix-org/complement` -> `tests/csapi/admin_test.go`](78c255edce/tests/csapi/admin_test.go) (added in https://github.com/matrix-org/complement/pull/322) in the Complement repo iteslf that tests the `/_synapse/admin/v1/send_server_notice` endpoint but it's a bit of an interesting case as [Dendrite also supports this endpoint](https://github.com/matrix-org/dendrite/pull/2180). I don't think it's good practice to continually shove in more and more Synapse-specific behavior into the Complement repo itself. We already have success with other out-of-repo tests for projects like the [SBG](b76b05b53e/complement_tests), [TI-Messenger Proxy](c8fa87fecc/complement), and our [Synapse Pro for small hosts](c2ea7eabf3/complement).
55 lines
1.9 KiB
Markdown
55 lines
1.9 KiB
Markdown
# Complement testing
|
|
|
|
Complement is a black box integration testing framework for Matrix homeservers. It
|
|
allows us to write end-to-end tests that interact with real Synapse homeservers to
|
|
ensure everything works at a holistic level.
|
|
|
|
|
|
## Setup
|
|
|
|
Nothing beyond a [normal Complement
|
|
setup](https://github.com/matrix-org/complement?tab=readme-ov-file#running) (just Go and
|
|
Docker).
|
|
|
|
|
|
## Running tests
|
|
|
|
Run tests from the [Complement](https://github.com/matrix-org/complement) repo:
|
|
|
|
```shell
|
|
# Run the tests
|
|
./scripts-dev/complement.sh
|
|
|
|
# To run a whole group of tests, you can specify part of the test path:
|
|
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
|
|
# To run a specific test, you can specify the whole name structure:
|
|
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate/Parallel/POST_/createRoom_makes_a_public_room
|
|
# Generally though, the `-run` parameter accepts regex patterns, so you can match however you like:
|
|
scripts-dev/complement.sh ./tests/... -run 'TestRoomCreate/Parallel/POST_/createRoom_makes_a_(.*)'
|
|
```
|
|
|
|
Typically, if you're developing the Synapse and Complement tests side-by-side, you will
|
|
run something like this:
|
|
|
|
```shell
|
|
# To run a specific test
|
|
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
|
|
```
|
|
|
|
|
|
### Running in-repo tests
|
|
|
|
In-repo Complement tests are tests that are vendored into this project. We use the
|
|
in-repo test suite to test Synapse specific behaviors like the admin API.
|
|
|
|
To run the in-repo Complement tests, use the `--in-repo` command line argument.
|
|
|
|
```shell
|
|
# Run only a specific test package.
|
|
# Note: test packages are relative to the `./complement` directory in this project
|
|
./scripts-dev/complement.sh --in-repo ./tests/...
|
|
|
|
# Similarly, you can also use `-run` to specify all or part of a specific test path to run
|
|
scripts-dev/complement.sh --in-repo ./tests/... -run TestIntraShardFederation
|
|
```
|