46d235cd52
This is useful so we can test Synapse specific behaviors like our admin API. (see docs in PR, `complement/README.md`) ``` COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo ``` Complement calls these ["out-of-repo"](78c255edce/OUT-OF-REPO-TESTS.md) tests but it's a bit of a misnomer once they're in your project. (just depends on the perspective) There has been [previous desire](https://github.com/element-hq/synapse/pull/19021#discussion_r2453442191) for this kind of thing but this is spawning from wanting to have some tests for our purge history admin API (https://github.com/element-hq/synapse-rust-apps/issues/430). There are some Sytest tests ([`matrix-org/sytest` -> `tests/48admin.pl#L91-L618`](1be04cce46/tests/48admin.pl (L91-L618))) for this already but I'd much rather work in Complement instead of Sytest. I'm wanting these tests to ensure that our new `event-cache` rust app for Synapse Pro doesn't break these kind of erasure features (https://github.com/element-hq/synapse-rust-apps/issues/366 and https://github.com/element-hq/synapse-rust-apps/issues/153). Interestingly, there is already [`matrix-org/complement` -> `tests/csapi/admin_test.go`](78c255edce/tests/csapi/admin_test.go) (added in https://github.com/matrix-org/complement/pull/322) in the Complement repo iteslf that tests the `/_synapse/admin/v1/send_server_notice` endpoint but it's a bit of an interesting case as [Dendrite also supports this endpoint](https://github.com/matrix-org/dendrite/pull/2180). I don't think it's good practice to continually shove in more and more Synapse-specific behavior into the Complement repo itself. We already have success with other out-of-repo tests for projects like the [SBG](b76b05b53e/complement_tests), [TI-Messenger Proxy](c8fa87fecc/complement), and our [Synapse Pro for small hosts](c2ea7eabf3/complement).
Complement testing
Complement is a black box integration testing framework for Matrix homeservers. It allows us to write end-to-end tests that interact with real Synapse homeservers to ensure everything works at a holistic level.
Setup
Nothing beyond a normal Complement setup (just Go and Docker).
Running tests
Run tests from the Complement repo:
# Run the tests
./scripts-dev/complement.sh
# To run a whole group of tests, you can specify part of the test path:
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
# To run a specific test, you can specify the whole name structure:
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate/Parallel/POST_/createRoom_makes_a_public_room
# Generally though, the `-run` parameter accepts regex patterns, so you can match however you like:
scripts-dev/complement.sh ./tests/... -run 'TestRoomCreate/Parallel/POST_/createRoom_makes_a_(.*)'
Typically, if you're developing the Synapse and Complement tests side-by-side, you will run something like this:
# To run a specific test
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
Running in-repo tests
In-repo Complement tests are tests that are vendored into this project. We use the in-repo test suite to test Synapse specific behaviors like the admin API.
To run the in-repo Complement tests, use the --in-repo command line argument.
# Run only a specific test package.
# Note: test packages are relative to the `./complement` directory in this project
./scripts-dev/complement.sh --in-repo ./tests/...
# Similarly, you can also use `-run` to specify all or part of a specific test path to run
scripts-dev/complement.sh --in-repo ./tests/... -run TestIntraShardFederation