kaslo/synapse
1
0
Fork 0
Code Wiki Activity

Merge branch 'madlittlemods/return-historical-events-in-order-from-backfill' into madlittlemods/optimize-msc2716-v1

Browse Source 
Conflicts:
	scripts-dev/complement.sh
This commit is contained in:
Eric Eastwood
2022-02-07 18:25:06 -06:00
parent 55ef667e96 b3fcffbd16
commit cb0ff0b0d5
253 changed files with 6285 additions and 3037 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.ci/scripts/test_old_deps.sh
View File

@@ -1,16 +1,18 @@
#!/usr/bin/env bash
# this script is run by GitHub Actions in a plain `bionic` container; it installs the
# this script is run by GitHub Actions in a plain `focal` container; it installs the
# minimal requirements for tox and hands over to the py3-old tox environment.
# Prevent tzdata from asking for user input
export DEBIAN_FRONTEND=noninteractive
set -ex
apt-get update
apt-get install -y python3 python3-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox
apt-get install -y python3 python3-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox libjpeg-dev libwebp-dev
export LANG="C.UTF-8"
# Prevent virtualenv from auto-updating pip to an incompatible version
export VIRTUALENV_NO_DOWNLOAD=1
exec tox -e py3-old,combine
exec tox -e py3-old

1
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@@ -8,6 +8,7 @@
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
* [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
* [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
(run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

14
.github/workflows/docker.yml vendored
View File

@@ -34,6 +34,8 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# TODO: consider using https://github.com/docker/metadata-action instead of this
# custom magic
- name: Calculate docker image tag
id: set-tag
run: |
@@ -53,18 +55,6 @@ jobs:
esac
echo "::set-output name=tag::$tag"
# for release builds, we want to get the amd64 image out asap, so first
# we do an amd64-only build, before following up with a multiarch build.
- name: Build and push amd64
uses: docker/build-push-action@v2
if: "${{ startsWith(github.ref, 'refs/tags/v') }}"
with:
push: true
labels: "gitsha1=${{ github.sha }}"
tags: "matrixdotorg/synapse:${{ steps.set-tag.outputs.tag }}"
file: "docker/Dockerfile"
platforms: linux/amd64
- name: Build and push all platforms
uses: docker/build-push-action@v2
with:

36
.github/workflows/tests.yml vendored
View File

@@ -141,7 +141,7 @@ jobs:
steps:
- uses: actions/checkout@v2
- name: Test with old deps
uses: docker://ubuntu:bionic # For old python and sqlite
uses: docker://ubuntu:focal # For old python and sqlite
with:
workdir: /github/workspace
entrypoint: .ci/scripts/test_old_deps.sh
@@ -213,15 +213,15 @@ jobs:
fail-fast: false
matrix:
include:
- sytest-tag: bionic
- sytest-tag: focal
- sytest-tag: bionic
- sytest-tag: focal
postgres: postgres
- sytest-tag: testing
postgres: postgres
- sytest-tag: bionic
- sytest-tag: focal
postgres: multi-postgres
workers: workers
@@ -323,17 +323,22 @@ jobs:
if: ${{ !failure() && !cancelled() }}
needs: linting-done
runs-on: ubuntu-latest
container:
# https://github.com/matrix-org/complement/blob/master/dockerfiles/ComplementCIBuildkite.Dockerfile
image: matrixdotorg/complement:latest
env:
CI: true
ports:
- 8448:8448
volumes:
- /var/run/docker.sock:/var/run/docker.sock
steps:
# The path is set via a file given by $GITHUB_PATH. We need both Go 1.17 and GOPATH on the path to run Complement.
# See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-system-path
- name: "Set Go Version"
run: |
# Add Go 1.17 to the PATH: see https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md#environment-variables-2
echo "$GOROOT_1_17_X64/bin" >> $GITHUB_PATH
# Add the Go path to the PATH: We need this so we can call gotestfmt
echo "~/go/bin" >> $GITHUB_PATH
- name: "Install Complement Dependencies"
run: |
sudo apt-get update && sudo apt-get install -y libolm3 libolm-dev
go get -v github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
- name: Run actions/checkout@v2 for synapse
uses: actions/checkout@v2
with:
@@ -376,8 +381,11 @@ jobs:
working-directory: complement/dockerfiles
# Run Complement
- run: set -o pipefail && go test -v -json -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
- run: |
set -o pipefail
go test -v -json -p 1 -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
shell: bash
name: Run Complement Tests
env:
COMPLEMENT_BASE_IMAGE: complement-synapse:latest
working-directory: complement

2
.github/workflows/twisted_trunk.yml vendored
View File

@@ -25,7 +25,7 @@ jobs:
- run: sudo apt-get -qq install xmlsec1
- uses: actions/setup-python@v2
with:
python-version: 3.6
python-version: 3.7
- run: .ci/patch_for_twisted_trunk.sh
- run: pip install tox
- run: tox -e py

2
.gitignore vendored
View File

@@ -52,5 +52,5 @@ __pycache__/
book/
# complement
/complement-master
/complement-*
/master.tar.gz

209
CHANGES.md
View File

@@ -1,8 +1,215 @@
Synapse 1.50.0rc1 (2022-01-05)
Synapse 1.52.0rc1 (2022-02-01)
==============================
Features
--------
- Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11621](https://github.com/matrix-org/synapse/issues/11621), [\#11788](https://github.com/matrix-org/synapse/issues/11788), [\#11789](https://github.com/matrix-org/synapse/issues/11789))
- Add an admin API to reset connection timeouts for remote server. ([\#11639](https://github.com/matrix-org/synapse/issues/11639))
- Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\#11658](https://github.com/matrix-org/synapse/issues/11658))
- Add a config flag to inhibit M_USER_IN_USE during registration. ([\#11743](https://github.com/matrix-org/synapse/issues/11743))
- Add a module callback to set username at registration. ([\#11790](https://github.com/matrix-org/synapse/issues/11790))
- Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\#11846](https://github.com/matrix-org/synapse/issues/11846))
Bugfixes
--------
- Include the bundled aggregations in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612))
- Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\#11767](https://github.com/matrix-org/synapse/issues/11767))
- Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\#11784](https://github.com/matrix-org/synapse/issues/11784))
- Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\#11798](https://github.com/matrix-org/synapse/issues/11798))
- Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\#11827](https://github.com/matrix-org/synapse/issues/11827))
Improved Documentation
----------------------
- Update pypi installation docs to indicate that we now support Python 3.10. ([\#11820](https://github.com/matrix-org/synapse/issues/11820))
- Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\#11821](https://github.com/matrix-org/synapse/issues/11821))
- Remove not needed old table of contents in documentation. ([\#11860](https://github.com/matrix-org/synapse/issues/11860))
- Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\#11861](https://github.com/matrix-org/synapse/issues/11861))
Deprecations and Removals
-------------------------
- Drop support for Python 3.6, which is EOL. ([\#11683](https://github.com/matrix-org/synapse/issues/11683))
- Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\#11843](https://github.com/matrix-org/synapse/issues/11843))
Internal Changes
----------------
- Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\#11792](https://github.com/matrix-org/synapse/issues/11792))
- Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\#11793](https://github.com/matrix-org/synapse/issues/11793))
- Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\#11794](https://github.com/matrix-org/synapse/issues/11794))
- Drop unused table `public_room_list_stream`. ([\#11795](https://github.com/matrix-org/synapse/issues/11795))
- Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\#11799](https://github.com/matrix-org/synapse/issues/11799), [\#11847](https://github.com/matrix-org/synapse/issues/11847))
- Docker: skip the initial amd64-only build and go straight to multiarch. ([\#11810](https://github.com/matrix-org/synapse/issues/11810))
- Run Complement on the Github Actions VM and not inside a Docker container. ([\#11811](https://github.com/matrix-org/synapse/issues/11811))
- Log module names at startup. ([\#11813](https://github.com/matrix-org/synapse/issues/11813))
- Improve type safety of bundled aggregations code. ([\#11815](https://github.com/matrix-org/synapse/issues/11815))
- Correct a type annotation in the event validation logic. ([\#11817](https://github.com/matrix-org/synapse/issues/11817), [\#11830](https://github.com/matrix-org/synapse/issues/11830))
- Minor updates and documentation for database schema delta files. ([\#11823](https://github.com/matrix-org/synapse/issues/11823))
- Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\#11834](https://github.com/matrix-org/synapse/issues/11834))
- Minor performance improvement in room state lookup. ([\#11836](https://github.com/matrix-org/synapse/issues/11836))
- Fix some indentation inconsistencies in the sample config. ([\#11838](https://github.com/matrix-org/synapse/issues/11838))
- Add type hints to `tests/rest/admin`. ([\#11851](https://github.com/matrix-org/synapse/issues/11851))
Synapse 1.51.0 (2022-01-25)
===========================
No significant changes since 1.51.0rc2.
Synapse 1.51.0 deprecates `webclient` listeners and non-HTTP(S) `web_client_location`s. Support for these will be removed in Synapse 1.53.0, at which point Synapse will not be capable of directly serving a web client for Matrix. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1510).
Synapse 1.51.0rc2 (2022-01-24)
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
Synapse 1.50.2 (2022-01-24)
===========================
This release includes the same bugfix as Synapse 1.51.0rc2.
Bugfixes
--------
- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
Synapse 1.51.0rc1 (2022-01-21)
==============================
Features
--------
- Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](https://github.com/matrix-org/synapse/issues/11561), [\#11749](https://github.com/matrix-org/synapse/issues/11749), [\#11757](https://github.com/matrix-org/synapse/issues/11757))
- Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11577](https://github.com/matrix-org/synapse/issues/11577))
- Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](https://github.com/matrix-org/synapse/issues/11672))
- Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](https://github.com/matrix-org/synapse/issues/11675), [\#11770](https://github.com/matrix-org/synapse/issues/11770))
Bugfixes
--------
- Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events
received over federation. ([\#11530](https://github.com/matrix-org/synapse/issues/11530))
- Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](https://github.com/matrix-org/synapse/issues/11587))
- Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](https://github.com/matrix-org/synapse/issues/11593))
- Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612), [\#11659](https://github.com/matrix-org/synapse/issues/11659), [\#11791](https://github.com/matrix-org/synapse/issues/11791))
- Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](https://github.com/matrix-org/synapse/issues/11667))
- Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](https://github.com/matrix-org/synapse/issues/11669))
- Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](https://github.com/matrix-org/synapse/issues/11695))
- Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](https://github.com/matrix-org/synapse/issues/11710), [\#11745](https://github.com/matrix-org/synapse/issues/11745))
- Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](https://github.com/matrix-org/synapse/issues/11737))
- Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](https://github.com/matrix-org/synapse/issues/11775))
- Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](https://github.com/matrix-org/synapse/issues/11786))
Improved Documentation
----------------------
- Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](https://github.com/matrix-org/synapse/issues/11686))
- Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](https://github.com/matrix-org/synapse/issues/11715))
- Document that the minimum supported PostgreSQL version is now 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725))
- Fix typo in demo docs: differnt. ([\#11735](https://github.com/matrix-org/synapse/issues/11735))
- Update room spec URL in config files. ([\#11739](https://github.com/matrix-org/synapse/issues/11739))
- Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](https://github.com/matrix-org/synapse/issues/11740))
- Update documentation for configuring login with Facebook. ([\#11755](https://github.com/matrix-org/synapse/issues/11755))
- Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](https://github.com/matrix-org/synapse/issues/11781))
Deprecations and Removals
-------------------------
- Remove the unstable `/send_relation` endpoint. ([\#11682](https://github.com/matrix-org/synapse/issues/11682))
- Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724))
- Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](https://github.com/matrix-org/synapse/issues/11576))
- **Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](https://github.com/matrix-org/synapse/issues/11774), [\#11783](https://github.com/matrix-org/synapse/issues/11783))**
Internal Changes
----------------
- Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](https://github.com/matrix-org/synapse/issues/11685))
- Use buildkit's cache feature to speed up docker builds. ([\#11691](https://github.com/matrix-org/synapse/issues/11691))
- Use `auto_attribs` and native type hints for attrs classes. ([\#11692](https://github.com/matrix-org/synapse/issues/11692), [\#11768](https://github.com/matrix-org/synapse/issues/11768))
- Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](https://github.com/matrix-org/synapse/issues/11693))
- Remove fallback code for Python 2. ([\#11699](https://github.com/matrix-org/synapse/issues/11699))
- Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic. ([\#11701](https://github.com/matrix-org/synapse/issues/11701))
- Add the option to write SQLite test dbs to disk when running tests. ([\#11702](https://github.com/matrix-org/synapse/issues/11702))
- Improve Complement test output for Gitub Actions. ([\#11707](https://github.com/matrix-org/synapse/issues/11707))
- Fix docstring on `add_account_data_for_user`. ([\#11716](https://github.com/matrix-org/synapse/issues/11716))
- Complement environment variable name change and update `.gitignore`. ([\#11718](https://github.com/matrix-org/synapse/issues/11718))
- Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](https://github.com/matrix-org/synapse/issues/11723))
- Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724), [\#11771](https://github.com/matrix-org/synapse/issues/11771))
- Minor efficiency improvements when inserting many values into the database. ([\#11742](https://github.com/matrix-org/synapse/issues/11742))
- Invite PR authors to give themselves credit in the changelog. ([\#11744](https://github.com/matrix-org/synapse/issues/11744))
- Add optional debugging to investigate [issue 8631](https://github.com/matrix-org/synapse/issues/8631). ([\#11760](https://github.com/matrix-org/synapse/issues/11760))
- Remove `log_function` utility function and its uses. ([\#11761](https://github.com/matrix-org/synapse/issues/11761))
- Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](https://github.com/matrix-org/synapse/issues/11765))
- Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](https://github.com/matrix-org/synapse/issues/11766))
- Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](https://github.com/matrix-org/synapse/issues/11776))
Synapse 1.50.1 (2022-01-18)
===========================
This release fixes a bug in Synapse 1.50.0 that could prevent clients from being able to connect to Synapse if the `webclient` resource was enabled. Further details are available in [this issue](https://github.com/matrix-org/synapse/issues/11763).
Bugfixes
--------
- Fix a bug introduced in Synapse 1.50.0rc1 that could cause Matrix clients to be unable to connect to Synapse instances with the `webclient` resource enabled. ([\#11764](https://github.com/matrix-org/synapse/issues/11764))
Synapse 1.50.0 (2022-01-18)
===========================
**This release contains a critical bug that may prevent clients from being able to connect.
As such, it is not recommended to upgrade to 1.50.0. Instead, please upgrade straight to
to 1.50.1. Further details are available in [this issue](https://github.com/matrix-org/synapse/issues/11763).**
Please note that we now only support Python 3.7+ and PostgreSQL 10+ (if applicable), because Python 3.6 and PostgreSQL 9.6 have reached end-of-life.
No significant changes since 1.50.0rc2.
Synapse 1.50.0rc2 (2022-01-14)
==============================
This release candidate fixes a federation-breaking regression introduced in Synapse 1.50.0rc1.
Bugfixes
--------
- Fix a bug introduced in Synapse v1.0.0 whereby some device list updates would not be sent to remote homeservers if there were too many to send at once. ([\#11729](https://github.com/matrix-org/synapse/issues/11729))
- Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could fail because too many EDUs were produced for device updates. ([\#11730](https://github.com/matrix-org/synapse/issues/11730))
Improved Documentation
----------------------
- Document that now the minimum supported PostgreSQL version is 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725))
Internal Changes
----------------
- Fix a typechecker problem related to our (ab)use of `nacl.signing.SigningKey`s. ([\#11714](https://github.com/matrix-org/synapse/issues/11714))
Synapse 1.50.0rc1 (2022-01-05)
==============================
Features
--------

1
changelog.d/11215.feature Normal file
View File

@@ -0,0 +1 @@
Add experimental support for sending to-device messages to application services, as specified by [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409). Disabled by default.

2
changelog.d/11530.bugfix
View File

@@ -1,2 +0,0 @@
Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events
received over federation.

1
changelog.d/11561.feature
View File

@@ -1 +0,0 @@
Add `track_puppeted_user_ips` config flag to track puppeted user IP addresses. This also includes them in monthly active user counts.

1
changelog.d/11587.bugfix
View File

@@ -1 +0,0 @@
Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices.

1
changelog.d/11593.bugfix
View File

@@ -1 +0,0 @@
Fix an error in to get federation status of a destination server even if no error has occurred. This admin API was new introduced in Synapse 1.49.0.

1
changelog.d/11612.misc
View File

@@ -1 +0,0 @@
Avoid database access in the JSON serialization process.

1
changelog.d/11615.misc Normal file
View File

@@ -0,0 +1 @@
Enhance user registration test helpers to make them more useful for tests involving Application Services and devices.

1
changelog.d/11616.misc Normal file
View File

@@ -0,0 +1 @@
Enhance user registration test helpers to make them more useful for tests involving Application Services and devices.

1
changelog.d/11655.feature Normal file
View File

@@ -0,0 +1 @@
Remove account data (including client config, push rules and ignored users) upon user deactivation.

1
changelog.d/11667.bugfix
View File

@@ -1 +0,0 @@
Fix `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0.

1
changelog.d/11672.feature
View File

@@ -1 +0,0 @@
Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room.

1
changelog.d/11682.removal
View File

@@ -1 +0,0 @@
Remove the unstable `/send_relation` endpoint.

1
changelog.d/11685.misc
View File

@@ -1 +0,0 @@
Run `pyupgrade --py37-plus --keep-percent-format` on Synapse.

1
changelog.d/11691.misc
View File

@@ -1 +0,0 @@
Use buildkit's cache feature to speed up docker builds.

1
changelog.d/11693.misc
View File

@@ -1 +0,0 @@
Remove debug logging for #4422, which has been closed since Synapse 0.99.

1
changelog.d/11695.bugfix
View File

@@ -1 +0,0 @@
Fix a bug where the only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0.

1
changelog.d/11699.misc
View File

@@ -1 +0,0 @@
Remove fallback code for Python 2.

1
changelog.d/11701.misc
View File

@@ -1 +0,0 @@
Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic.

1
changelog.d/11702.misc
View File

@@ -1 +0,0 @@
Add the option to write sqlite test dbs to disk when running tests.

1
changelog.d/11707.misc
View File

@@ -1 +0,0 @@
Improve Complement test output for Gitub Actions.

1
changelog.d/11714.misc
View File

@@ -1 +0,0 @@
Fix a typechecker problem related to our (ab)use of `nacl.signing.SigningKey`s.

1
changelog.d/11715.doc
View File

@@ -1 +0,0 @@
Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide.

1
changelog.d/11716.misc
View File

@@ -1 +0,0 @@
Fix docstring on `add_account_data_for_user`.

1
changelog.d/11718.misc
View File

@@ -1 +0,0 @@
Complement environment variable name change and update `.gitignore`.

1
changelog.d/11725.doc
View File

@@ -1 +0,0 @@
Document that now the minimum supported PostgreSQL version is 10.

1
changelog.d/11832.misc Normal file
View File

@@ -0,0 +1 @@
Fix type errors introduced by new annotations in the Prometheus Client library.

1
changelog.d/11859.doc Normal file
View File

@@ -0,0 +1 @@
Fix typo in User Admin API: unpind -> unbind.

1
changelog.d/11862.doc Normal file
View File

@@ -0,0 +1 @@
Document images returned by the User List Media Admin API can include those generated by URL previews.

5
changelog.d/11867.feature Normal file
View File

@@ -0,0 +1,5 @@
Stabilize [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
Client implementations using `m.login.registration_token` should switch to the stable identifiers:
* `org.matrix.msc3231.login.registration_token` in query parameters and request/response bodies becomes `m.login.registration_token`.
* `/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity` becomes `/_matrix/client/v1/register/m.login.registration_token/validity`.

1
changelog.d/11868.feature Normal file
View File

@@ -0,0 +1 @@
Allow modules to retrieve the current instance's server name and worker name.

1
changelog.d/11869.misc Normal file
View File

@@ -0,0 +1 @@
Ensure that `opentracing` scopes are activated and closed at the right time.

1
changelog.d/11870.misc Normal file
View File

@@ -0,0 +1 @@
Improve opentracing for incoming federation requests.

1
changelog.d/11876.misc Normal file
View File

@@ -0,0 +1 @@
Improve internal docstrings in `synapse.util.caches`.

1
changelog.d/11878.misc Normal file
View File

@@ -0,0 +1 @@
Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes.

1
changelog.d/11880.misc Normal file
View File

@@ -0,0 +1 @@
Convert `ApplicationServiceTestCase` to use `simple_async_mock`.

1
changelog.d/11884.misc Normal file
View File

@@ -0,0 +1 @@
Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled.

1
changelog.d/11888.misc Normal file
View File

@@ -0,0 +1 @@
Disable coverage calculation for olddeps build.

1
changelog.d/11890.bugfix Normal file
View File

@@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.51.0rc1 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`.

1
changelog.d/11892.feature Normal file
View File

@@ -0,0 +1 @@
Use a dedicated configurable rate limiter for 3PID invites.

1
changelog.d/11895.removal Normal file
View File

@@ -0,0 +1 @@
Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error.

1
changelog.d/11907.doc Normal file
View File

@@ -0,0 +1 @@
Remove outdated MSC1711 FAQ document.

1
changelog.d/11910.misc Normal file
View File

@@ -0,0 +1 @@
Run Complement tests sequentially.

16
contrib/prometheus/consoles/synapse.html
View File

@@ -92,22 +92,6 @@ new PromConsole.Graph({
})
</script>
<h3>Pending calls per tick</h3>
<div id="reactor_pending_calls"></div>
<script>
new PromConsole.Graph({
node: document.querySelector("#reactor_pending_calls"),
expr: "rate(python_twisted_reactor_pending_calls_sum[30s]) / rate(python_twisted_reactor_pending_calls_count[30s])",
name: "[[job]]-[[index]]",
min: 0,
renderer: "line",
height: 150,
yAxisFormatter: PromConsole.NumberFormatter.humanize,
yHoverFormatter: PromConsole.NumberFormatter.humanize,
yTitle: "Pending Calls"
})
</script>
<h1>Storage</h1>
<h3>Queries</h3>

48
debian/changelog vendored
View File

@@ -1,3 +1,51 @@
matrix-synapse-py3 (1.52.0~rc1) stable; urgency=medium
* New synapse release 1.52.0~rc1.
-- Synapse Packaging team <packages@matrix.org> Tue, 01 Feb 2022 11:04:09 +0000
matrix-synapse-py3 (1.51.0) stable; urgency=medium
* New synapse release 1.51.0.
-- Synapse Packaging team <packages@matrix.org> Tue, 25 Jan 2022 11:28:51 +0000
matrix-synapse-py3 (1.51.0~rc2) stable; urgency=medium
* New synapse release 1.51.0~rc2.
-- Synapse Packaging team <packages@matrix.org> Mon, 24 Jan 2022 12:25:00 +0000
matrix-synapse-py3 (1.51.0~rc1) stable; urgency=medium
* New synapse release 1.51.0~rc1.
-- Synapse Packaging team <packages@matrix.org> Fri, 21 Jan 2022 10:46:02 +0000
matrix-synapse-py3 (1.50.2) stable; urgency=medium
* New synapse release 1.50.2.
-- Synapse Packaging team <packages@matrix.org> Mon, 24 Jan 2022 13:37:11 +0000
matrix-synapse-py3 (1.50.1) stable; urgency=medium
* New synapse release 1.50.1.
-- Synapse Packaging team <packages@matrix.org> Tue, 18 Jan 2022 16:06:26 +0000
matrix-synapse-py3 (1.50.0) stable; urgency=medium
* New synapse release 1.50.0.
-- Synapse Packaging team <packages@matrix.org> Tue, 18 Jan 2022 10:40:38 +0000
matrix-synapse-py3 (1.50.0~rc2) stable; urgency=medium
* New synapse release 1.50.0~rc2.
-- Synapse Packaging team <packages@matrix.org> Fri, 14 Jan 2022 11:18:06 +0000
matrix-synapse-py3 (1.50.0~rc1) stable; urgency=medium
* New synapse release 1.50.0~rc1.

2
demo/README
View File

@@ -22,5 +22,5 @@ Logs and sqlitedb will be stored in demo/808{0,1,2}.{log,db}
Also note that when joining a public room on a differnt HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.
Also note that when joining a public room on a different HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.

2
docker/Dockerfile-pgtests
View File

@@ -1,6 +1,6 @@
# Use the Sytest image that comes with a lot of the build dependencies
# pre-installed
FROM matrixdotorg/sytest:bionic
FROM matrixdotorg/sytest:focal
# The Sytest image doesn't come with python, so install that
RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip

2
docker/run_pg_tests.sh
View File

@@ -16,4 +16,4 @@ sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/dat
# Run the tests
cd /src
export TRIAL_FLAGS="-j 4"
tox --workdir=./.tox-pg-container -e py36-postgres "$@"
tox --workdir=./.tox-pg-container -e py37-postgres "$@"

335
docs/MSC1711_certificates_FAQ.md
View File

@@ -1,335 +0,0 @@
# MSC1711 Certificates FAQ
## Historical Note
This document was originally written to guide server admins through the upgrade
path towards Synapse 1.0. Specifically,
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md)
required that all servers present valid TLS certificates on their federation
API. Admins were encouraged to achieve compliance from version 0.99.0 (released
in February 2019) ahead of version 1.0 (released June 2019) enforcing the
certificate checks.
Much of what follows is now outdated since most admins will have already
upgraded, however it may be of use to those with old installs returning to the
project.
If you are setting up a server from scratch you almost certainly should look at
the [installation guide](setup/installation.md) instead.
## Introduction
The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It
supports the r0.1 release of the server to server specification, but is
compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well
as post-r0.1 behaviour, in order to allow for a smooth upgrade across the
federation.
The most important thing to know is that Synapse 1.0.0 will require a valid TLS
certificate on federation endpoints. Self signed certificates will not be
sufficient.
Synapse 0.99.0 makes it easy to configure TLS certificates and will
interoperate with both >= 1.0.0 servers as well as existing servers yet to
upgrade.
**It is critical that all admins upgrade to 0.99.0 and configure a valid TLS
certificate.** Admins will have 1 month to do so, after which 1.0.0 will be
released and those servers without a valid certificate will not longer be able
to federate with >= 1.0.0 servers.
Full details on how to carry out this configuration change is given
[below](#configuring-certificates-for-compatibility-with-synapse-100). A
timeline and some frequently asked questions are also given below.
For more details and context on the release of the r0.1 Server/Server API and
imminent Matrix 1.0 release, you can also see our
[main talk from FOSDEM 2019](https://matrix.org/blog/2019/02/04/matrix-at-fosdem-2019/).
## Contents
* Timeline
* Configuring certificates for compatibility with Synapse 1.0
* FAQ
* Synapse 0.99.0 has just been released, what do I need to do right now?
* How do I upgrade?
* What will happen if I do not set up a valid federation certificate
immediately?
* What will happen if I do nothing at all?
* When do I need a SRV record or .well-known URI?
* Can I still use an SRV record?
* I have created a .well-known URI. Do I still need an SRV record?
* It used to work just fine, why are you breaking everything?
* Can I manage my own certificates rather than having Synapse renew
certificates itself?
* Do you still recommend against using a reverse proxy on the federation port?
* Do I still need to give my TLS certificates to Synapse if I am using a
reverse proxy?
* Do I need the same certificate for the client and federation port?
* How do I tell Synapse to reload my keys/certificates after I replace them?
## Timeline
**5th Feb 2019 - Synapse 0.99.0 is released.**
All server admins are encouraged to upgrade.
0.99.0:
- provides support for ACME to make setting up Let's Encrypt certs easy, as
well as .well-known support.
- does not enforce that a valid CA cert is present on the federation API, but
rather makes it easy to set one up.
- provides support for .well-known
Admins should upgrade and configure a valid CA cert. Homeservers that require a
.well-known entry (see below), should retain their SRV record and use it
alongside their .well-known record.
**10th June 2019 - Synapse 1.0.0 is released**
1.0.0 is scheduled for release on 10th June. In
accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html)
1.0.0 will enforce certificate validity. This means that any homeserver without a
valid certificate after this point will no longer be able to federate with
1.0.0 servers.
## Configuring certificates for compatibility with Synapse 1.0.0
### If you do not currently have an SRV record
In this case, your `server_name` points to the host where your Synapse is
running. There is no need to create a `.well-known` URI or an SRV record, but
you will need to give Synapse a valid, signed, certificate.
### If you do have an SRV record currently
If you are using an SRV record, your matrix domain (`server_name`) may not
point to the same host that your Synapse is running on (the 'target
domain'). (If it does, you can follow the recommendation above; otherwise, read
on.)
Let's assume that your `server_name` is `example.com`, and your Synapse is
hosted at a target domain of `customer.example.net`. Currently you should have
an SRV record which looks like:
```
_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net.
```
In this situation, you have three choices for how to proceed:
#### Option 1: give Synapse a certificate for your matrix domain
Synapse 1.0 will expect your server to present a TLS certificate for your
`server_name` (`example.com` in the above example). You can achieve this by acquiring a
certificate for the `server_name` yourself (for example, using `certbot`), and giving it
and the key to Synapse via `tls_certificate_path` and `tls_private_key_path`.
#### Option 2: run Synapse behind a reverse proxy
If you have an existing reverse proxy set up with correct TLS certificates for
your domain, you can simply route all traffic through the reverse proxy by
updating the SRV record appropriately (or removing it, if the proxy listens on
8448).
See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
reverse proxy.
#### Option 3: add a .well-known file to delegate your matrix traffic
This will allow you to keep Synapse on a separate domain, without having to
give it a certificate for the matrix domain.
You can do this with a `.well-known` file as follows:
1. Keep the SRV record in place - it is needed for backwards compatibility
with Synapse 0.34 and earlier.
2. Give Synapse a certificate corresponding to the target domain
(`customer.example.net` in the above example). You can do this by acquire a
certificate for the target domain and giving it to Synapse via `tls_certificate_path`
and `tls_private_key_path`.
3. Restart Synapse to ensure the new certificate is loaded.
4. Arrange for a `.well-known` file at
`https://<server_name>/.well-known/matrix/server` with contents:
```json
{"m.server": "<target server name>"}
```
where the target server name is resolved as usual (i.e. SRV lookup, falling
back to talking to port 8448).
In the above example, where synapse is listening on port 8000,
`https://example.com/.well-known/matrix/server` should have `m.server` set to one of:
1. `customer.example.net` ─ with a SRV record on
`_matrix._tcp.customer.example.com` pointing to port 8000, or:
2. `customer.example.net` ─ updating synapse to listen on the default port
8448, or:
3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy
on `customer.example.net:8000` it correctly handles HTTP requests with
Host header set to `customer.example.net:8000`.
## FAQ
### Synapse 0.99.0 has just been released, what do I need to do right now?
Upgrade as soon as you can in preparation for Synapse 1.0.0, and update your
TLS certificates as [above](#configuring-certificates-for-compatibility-with-synapse-100).
### What will happen if I do not set up a valid federation certificate immediately?
Nothing initially, but once 1.0.0 is in the wild it will not be possible to
federate with 1.0.0 servers.
### What will happen if I do nothing at all?
If the admin takes no action at all, and remains on a Synapse < 0.99.0 then the
homeserver will be unable to federate with those who have implemented
.well-known. Then, as above, once the month upgrade window has expired the
homeserver will not be able to federate with any Synapse >= 1.0.0
### When do I need a SRV record or .well-known URI?
If your homeserver listens on the default federation port (8448), and your
`server_name` points to the host that your homeserver runs on, you do not need an
SRV record or `.well-known/matrix/server` URI.
For instance, if you registered `example.com` and pointed its DNS A record at a
fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host,
giving it a server_name of `example.com`, and it would automatically generate a
valid TLS certificate for you via Let's Encrypt and no SRV record or
`.well-known` URI would be needed.
This is the common case, although you can add an SRV record or
`.well-known/matrix/server` URI for completeness if you wish.
**However**, if your server does not listen on port 8448, or if your `server_name`
does not point to the host that your homeserver runs on, you will need to let
other servers know how to find it.
In this case, you should see ["If you do have an SRV record
currently"](#if-you-do-have-an-srv-record-currently) above.
### Can I still use an SRV record?
Firstly, if you didn't need an SRV record before (because your server is
listening on port 8448 of your server_name), you certainly don't need one now:
the defaults are still the same.
If you previously had an SRV record, you can keep using it provided you are
able to give Synapse a TLS certificate corresponding to your server name. For
example, suppose you had the following SRV record, which directs matrix traffic
for example.com to matrix.example.com:443:
```
_matrix._tcp.example.com. IN SRV 10 5 443 matrix.example.com
```
In this case, Synapse must be given a certificate for example.com - or be
configured to acquire one from Let's Encrypt.
If you are unable to give Synapse a certificate for your server_name, you will
also need to use a .well-known URI instead. However, see also "I have created a
.well-known URI. Do I still need an SRV record?".
### I have created a .well-known URI. Do I still need an SRV record?
As of Synapse 0.99, Synapse will first check for the existence of a `.well-known`
URI and follow any delegation it suggests. It will only then check for the
existence of an SRV record.
That means that the SRV record will often be redundant. However, you should
remember that there may still be older versions of Synapse in the federation
which do not understand `.well-known` URIs, so if you removed your SRV record you
would no longer be able to federate with them.
It is therefore best to leave the SRV record in place for now. Synapse 0.34 and
earlier will follow the SRV record (and not care about the invalid
certificate). Synapse 0.99 and later will follow the .well-known URI, with the
correct certificate chain.
### It used to work just fine, why are you breaking everything?
We have always wanted Matrix servers to be as easy to set up as possible, and
so back when we started federation in 2014 we didn't want admins to have to go
through the cumbersome process of buying a valid TLS certificate to run a
server. This was before Let's Encrypt came along and made getting a free and
valid TLS certificate straightforward. So instead, we adopted a system based on
[Perspectives](https://en.wikipedia.org/wiki/Convergence_(SSL)): an approach
where you check a set of "notary servers" (in practice, homeservers) to vouch
for the validity of a certificate rather than having it signed by a CA. As long
as enough different notaries agree on the certificate's validity, then it is
trusted.
However, in practice this has never worked properly. Most people only use the
default notary server (matrix.org), leading to inadvertent centralisation which
we want to eliminate. Meanwhile, we never implemented the full consensus
algorithm to query the servers participating in a room to determine consensus
on whether a given certificate is valid. This is fiddly to get right
(especially in face of sybil attacks), and we found ourselves questioning
whether it was worth the effort to finish the work and commit to maintaining a
secure certificate validation system as opposed to focusing on core Matrix
development.
Meanwhile, Let's Encrypt came along in 2016, and put the final nail in the
coffin of the Perspectives project (which was already pretty dead). So, the
Spec Core Team decided that a better approach would be to mandate valid TLS
certificates for federation alongside the rest of the Web. More details can be
found in
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
This results in a breaking change, which is disruptive, but absolutely critical
for the security model. However, the existence of Let's Encrypt as a trivial
way to replace the old self-signed certificates with valid CA-signed ones helps
smooth things over massively, especially as Synapse can now automate Let's
Encrypt certificate generation if needed.
### Can I manage my own certificates rather than having Synapse renew certificates itself?
Yes, you are welcome to manage your certificates yourself. Synapse will only
attempt to obtain certificates from Let's Encrypt if you configure it to do
so.The only requirement is that there is a valid TLS cert present for
federation end points.
### Do you still recommend against using a reverse proxy on the federation port?
We no longer actively recommend against using a reverse proxy. Many admins will
find it easier to direct federation traffic to a reverse proxy and manage their
own TLS certificates, and this is a supported configuration.
See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
reverse proxy.
### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy?
Practically speaking, this is no longer necessary.
If you are using a reverse proxy for all of your TLS traffic, then you can set
`no_tls: True`. In that case, the only reason Synapse needs the certificate is
to populate a legacy 'tls_fingerprints' field in the federation API. This is
ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will
check it is when attempting to fetch the server keys - and generally this is
delegated via `matrix.org`, which is on 0.99.0.
However, there is a bug in Synapse 0.99.0
[4554](<https://github.com/matrix-org/synapse/issues/4554>) which prevents
Synapse from starting if you do not give it a TLS certificate. To work around
this, you can give it any TLS certificate at all. This will be fixed soon.
### Do I need the same certificate for the client and federation port?
No. There is nothing stopping you from using different certificates,
particularly if you are using a reverse proxy. However, Synapse will use the
same certificate on any ports where TLS is configured.
### How do I tell Synapse to reload my keys/certificates after I replace them?
Synapse will reload the keys and certificates when it receives a SIGHUP - for
example `kill -HUP $(cat homeserver.pid)`. Alternatively, simply restart
Synapse, though this will result in downtime while it restarts.

3
docs/SUMMARY.md
View File

@@ -13,7 +13,6 @@
# Upgrading
- [Upgrading between Synapse Versions](upgrade.md)
- [Upgrading from pre-Synapse 1.0](MSC1711_certificates_FAQ.md)
# Usage
- [Federation](federate.md)
@@ -72,7 +71,7 @@
- [Understanding Synapse Through Grafana Graphs](usage/administration/understanding_synapse_through_grafana_graphs.md)
- [Useful SQL for Admins](usage/administration/useful_sql_for_admins.md)
- [Database Maintenance Tools](usage/administration/database_maintenance_tools.md)
- [State Groups](usage/administration/state_groups.md)
- [State Groups](usage/administration/state_groups.md)
- [Request log format](usage/administration/request_log.md)
- [Admin FAQ](usage/administration/admin_faq.md)
- [Scripts]()

3
docs/admin_api/account_validity.md
View File

@@ -4,6 +4,9 @@ This API allows a server administrator to manage the validity of an account. To
use it, you must enable the account validity feature (under
`account_validity`) in Synapse's configuration.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
## Renew account
This API extends the validity of an account by as much time as configured in the

6
docs/admin_api/delete_group.md
View File

@@ -4,11 +4,11 @@ This API lets a server admin delete a local group. Doing so will kick all
users out of the group so that their clients will correctly handle the group
being deleted.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
The API is:
```
POST /_synapse/admin/v1/delete_group/<group_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).

7
docs/admin_api/event_reports.md
View File

@@ -2,12 +2,13 @@
This API returns information about reported events.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
The api is:
```
GET /_synapse/admin/v1/event_reports?from=0&limit=10
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).
It returns a JSON body like the following:
@@ -94,8 +95,6 @@ The api is:
```
GET /_synapse/admin/v1/event_reports/<report_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).
It returns a JSON body like the following:

28
docs/admin_api/media_admin_api.md
View File

@@ -1,24 +1,13 @@
# Contents
- [Querying media](#querying-media)
* [List all media in a room](#list-all-media-in-a-room)
* [List all media uploaded by a user](#list-all-media-uploaded-by-a-user)
- [Quarantine media](#quarantine-media)
* [Quarantining media by ID](#quarantining-media-by-id)
* [Remove media from quarantine by ID](#remove-media-from-quarantine-by-id)
* [Quarantining media in a room](#quarantining-media-in-a-room)
* [Quarantining all media of a user](#quarantining-all-media-of-a-user)
* [Protecting media from being quarantined](#protecting-media-from-being-quarantined)
* [Unprotecting media from being quarantined](#unprotecting-media-from-being-quarantined)
- [Delete local media](#delete-local-media)
* [Delete a specific local media](#delete-a-specific-local-media)
* [Delete local media by date or size](#delete-local-media-by-date-or-size)
* [Delete media uploaded by a user](#delete-media-uploaded-by-a-user)
- [Purge Remote Media API](#purge-remote-media-api)
# Querying media
These APIs allow extracting media information from the homeserver.
Details about the format of the `media_id` and storage of the media in the file system
are documented under [media repository](../media_repository.md).
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
## List all media in a room
This API gets a list of known media in a room.
@@ -28,8 +17,6 @@ The API is:
```
GET /_synapse/admin/v1/room/<room_id>/media
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).
The API returns a JSON body like the following:
```json
@@ -317,8 +304,5 @@ The following fields are returned in the JSON response body:
* `deleted`: integer - The number of media items successfully deleted
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).
If the user re-requests purged remote media, synapse will re-request the media
from the originating server.

9
docs/admin_api/purge_history_api.md
View File

@@ -10,15 +10,15 @@ paginate further back in the room from the point being purged from.
Note that Synapse requires at least one message in each room, so it will never
delete the last message in a room.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
The API is:
```
POST /_synapse/admin/v1/purge_history/<room_id>[/<event_id>]
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
By default, events sent by local users are not deleted, as they may represent
the only copies of this content in existence. (Events sent by remote users are
deleted.)
@@ -57,9 +57,6 @@ It is possible to poll for updates on recent purges with a second API;
GET /_synapse/admin/v1/purge_history_status/<purge_id>
```
Again, you will need to authenticate by providing an `access_token` for a
server admin.
This API returns a JSON body like the following:
```json

6
docs/admin_api/room_membership.md
View File

@@ -5,6 +5,9 @@ to a room with a given `room_id_or_alias`. You can only modify the membership of
local users. The server administrator must be in the room and have permission to
invite users.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
## Parameters
The following parameters are available:
@@ -23,9 +26,6 @@ POST /_synapse/admin/v1/join/<room_id_or_alias>
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: see [Admin API](../usage/administration/admin_api).
Response:
```json

21
docs/admin_api/rooms.md
View File

@@ -1,24 +1,12 @@
# Contents
- [List Room API](#list-room-api)
- [Room Details API](#room-details-api)
- [Room Members API](#room-members-api)
- [Room State API](#room-state-api)
- [Block Room API](#block-room-api)
- [Delete Room API](#delete-room-api)
* [Version 1 (old version)](#version-1-old-version)
* [Version 2 (new version)](#version-2-new-version)
* [Status of deleting rooms](#status-of-deleting-rooms)
* [Undoing room shutdowns](#undoing-room-shutdowns)
- [Make Room Admin API](#make-room-admin-api)
- [Forward Extremities Admin API](#forward-extremities-admin-api)
- [Event Context API](#event-context-api)
# List Room API
The List Room admin API allows server admins to get a list of rooms on their
server. There are various parameters available that allow for filtering and
sorting the returned list. This API supports pagination.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
**Parameters**
The following query parameters are available:
@@ -493,9 +481,6 @@ several minutes or longer.
The local server will only have the power to move local user and room aliases to
the new room. Users on other servers will be unaffected.
To use it, you will need to authenticate by providing an ``access_token`` for a
server admin: see [Admin API](../usage/administration/admin_api).
## Version 1 (old version)
This version works synchronously. That means you only get the response once the server has

6
docs/admin_api/statistics.md
View File

@@ -3,15 +3,15 @@
Returns information about all local media usage of users. Gives the
possibility to filter them by time and user.
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
The API is:
```
GET /_synapse/admin/v1/statistics/users/media
```
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
A response body like the following is returned:
```json

99
docs/admin_api/user_admin_api.md
View File

@@ -1,5 +1,8 @@
# User Admin API
To use it, you will need to authenticate by providing an `access_token`
for a server admin: see [Admin API](../usage/administration/admin_api).
## Query User Account
This API returns information about a specific user account.
@@ -10,14 +13,12 @@ The api is:
GET /_synapse/admin/v2/users/<user_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
It returns a JSON body like the following:
```json
```jsonc
{
"displayname": "User",
"name": "@user:example.com",
"displayname": "User", // can be null if not set
"threepids": [
{
"medium": "email",
@@ -32,11 +33,11 @@ It returns a JSON body like the following:
"validated_at": 1586458409743
}
],
"avatar_url": "<avatar_url>",
"avatar_url": "<avatar_url>", // can be null if not set
"is_guest": 0,
"admin": 0,
"deactivated": 0,
"shadow_banned": 0,
"password_hash": "$2b$12$p9B4GkqYdRTPGD",
"creation_ts": 1560432506,
"appservice_id": null,
"consent_server_notice_sent": null,
@@ -103,9 +104,6 @@ with a body of:
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
Returns HTTP status code:
- `201` - When a new user object was created.
- `200` - When a user was modified.
@@ -155,9 +153,6 @@ By default, the response is ordered by ascending user ID.
GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -277,9 +272,6 @@ GET /_matrix/client/r0/admin/whois/<userId>
See also: [Client Server
API Whois](https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid).
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
It returns a JSON body like the following:
```json
@@ -334,15 +326,12 @@ with a body of:
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
The erase parameter is optional and defaults to `false`.
An empty body may be passed for backwards compatibility.
The following actions are performed when deactivating an user:
- Try to unpind 3PIDs from the identity server
- Try to unbind 3PIDs from the identity server
- Remove all 3PIDs from the homeserver
- Delete all devices and E2EE keys
- Delete all access tokens
@@ -352,6 +341,11 @@ The following actions are performed when deactivating an user:
- Remove the user from the user directory
- Reject all pending invites
- Remove all account validity information related to the user
- Remove the arbitrary data store known as *account data*. For example, this includes:
- list of ignored users;
- push rules;
- secret storage keys; and
- cross-signing keys.
The following additional actions are performed during deactivation if `erase`
is set to `true`:
@@ -365,7 +359,6 @@ The following actions are **NOT** performed. The list may be incomplete.
- Remove mappings of SSO IDs
- [Delete media uploaded](#delete-media-uploaded-by-a-user) by user (included avatar images)
- Delete sent and received messages
- Delete E2E cross-signing keys
- Remove the user's creation (registration) timestamp
- [Remove rate limit overrides](#override-ratelimiting-for-users)
- Remove from monthly active users
@@ -389,9 +382,6 @@ with a body of:
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
The parameter `new_password` is required.
The parameter `logout_devices` is optional and defaults to `true`.
@@ -404,9 +394,6 @@ The api is:
GET /_synapse/admin/v1/users/<user_id>/admin
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -434,10 +421,6 @@ with a body of:
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
## List room memberships of a user
Gets a list of all `room_id` that a specific `user_id` is member.
@@ -448,9 +431,6 @@ The API is:
GET /_synapse/admin/v1/users/<user_id>/joined_rooms
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -559,6 +539,11 @@ The following fields are returned in the JSON response body:
### List media uploaded by a user
Gets a list of all local media that a specific `user_id` has created.
These are media that the user has uploaded themselves
([local media](../media_repository.md#local-media)), as well as
[URL preview images](../media_repository.md#url-previews) requested by the user if the
[feature is enabled](../development/url_previews.md).
By default, the response is ordered by descending creation date and ascending media ID.
The newest media is on top. You can change the order with parameters
`order_by` and `dir`.
@@ -569,9 +554,6 @@ The API is:
GET /_synapse/admin/v1/users/<user_id>/media
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -658,7 +640,9 @@ The following fields are returned in the JSON response body:
Media objects contain the following fields:
- `created_ts` - integer - Timestamp when the content was uploaded in ms.
- `last_access_ts` - integer - Timestamp when the content was last accessed in ms.
- `media_id` - string - The id used to refer to the media.
- `media_id` - string - The id used to refer to the media. Details about the format
are documented under
[media repository](../media_repository.md).
- `media_length` - integer - Length of the media in bytes.
- `media_type` - string - The MIME-type of the media.
- `quarantined_by` - string - The user ID that initiated the quarantine request
@@ -686,9 +670,6 @@ The API is:
DELETE /_synapse/admin/v1/users/<user_id>/media
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -761,9 +742,6 @@ The API is:
GET /_synapse/admin/v2/users/<user_id>/devices
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -829,9 +807,6 @@ POST /_synapse/admin/v2/users/<user_id>/delete_devices
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
**Parameters**
@@ -853,9 +828,6 @@ The API is:
GET /_synapse/admin/v2/users/<user_id>/devices/<device_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -901,9 +873,6 @@ PUT /_synapse/admin/v2/users/<user_id>/devices/<device_id>
}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
**Parameters**
@@ -930,9 +899,6 @@ DELETE /_synapse/admin/v2/users/<user_id>/devices/<device_id>
{}
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
**Parameters**
@@ -951,9 +917,6 @@ The API is:
GET /_synapse/admin/v1/users/<user_id>/pushers
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -1048,9 +1011,6 @@ To un-shadow-ban a user the API is:
DELETE /_synapse/admin/v1/users/<user_id>/shadow_ban
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned in both cases.
**Parameters**
@@ -1073,9 +1033,6 @@ The API is:
GET /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -1115,9 +1072,6 @@ The API is:
POST /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
```json
@@ -1160,9 +1114,6 @@ The API is:
DELETE /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
```json
@@ -1191,7 +1142,5 @@ The API is:
GET /_synapse/admin/v1/username_available?username=$localpart
```
The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.
To use it, you will need to authenticate by providing an `access_token` for a
server admin: [Admin API](../usage/administration/admin_api)
The request and response format is the same as the
[/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.

2
docs/admin_api/version_api.md
View File

@@ -16,6 +16,6 @@ It returns a JSON body like the following:
```json
{
"server_version": "0.99.2rc1 (b=develop, abcdef123)",
"python_version": "3.6.8"
"python_version": "3.7.8"
}
```

7
docs/development/contributing_guide.md
View File

@@ -20,7 +20,9 @@ recommended for development. More information about WSL can be found at
<https://docs.microsoft.com/en-us/windows/wsl/install>. Running Synapse natively
on Windows is not officially supported.
The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://wiki.python.org/moin/BeginnersGuide/Download).
The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://www.python.org/downloads/). Your Python also needs support for [virtual environments](https://docs.python.org/3/library/venv.html). This is usually built-in, but some Linux distributions like Debian and Ubuntu split it out into its own package. Running `sudo apt install python3-venv` should be enough.
Synapse can connect to PostgreSQL via the [psycopg2](https://pypi.org/project/psycopg2/) Python library. Building this library from source requires access to PostgreSQL's C header files. On Debian or Ubuntu Linux, these can be installed with `sudo apt install libpq-dev`.
The source code of Synapse is hosted on GitHub. You will also need [a recent version of git](https://github.com/git-guides/install-git).
@@ -53,6 +55,7 @@ setup a *virtualenv*, as follows:
cd path/where/you/have/cloned/the/repository
python3 -m venv ./env
source ./env/bin/activate
pip install wheel
pip install -e ".[all,dev]"
pip install tox
```
@@ -114,7 +117,7 @@ The linters look at your code and do two things:
- ensure that your code follows the coding style adopted by the project;
- catch a number of errors in your code.
They're pretty fast, don't hesitate!
The linter takes no time at all to run as soon as you've [downloaded the dependencies into your python virtual environment](#4-install-the-dependencies).
```sh
source ./env/bin/activate

54
docs/development/database_schema.md
View File

@@ -96,6 +96,60 @@ Ensure postgres is installed, then run:
NB at the time of writing, this script predates the split into separate `state`/`main`
databases so will require updates to handle that correctly.
## Delta files
Delta files define the steps required to upgrade the database from an earlier version.
They can be written as either a file containing a series of SQL statements, or a Python
module.
Synapse remembers which delta files it has applied to a database (they are stored in the
`applied_schema_deltas` table) and will not re-apply them (even if a given file is
subsequently updated).
Delta files should be placed in a directory named `synapse/storage/schema/<database>/delta/<version>/`.
They are applied in alphanumeric order, so by convention the first two characters
of the filename should be an integer such as `01`, to put the file in the right order.
### SQL delta files
These should be named `*.sql`, or — for changes which should only be applied for a
given database engine — `*.sql.posgres` or `*.sql.sqlite`. For example, a delta which
adds a new column to the `foo` table might be called `01add_bar_to_foo.sql`.
Note that our SQL parser is a bit simple - it understands comments (`--` and `/*...*/`),
but complex statements which require a `;` in the middle of them (such as `CREATE
TRIGGER`) are beyond it and you'll have to use a Python delta file.
### Python delta files
For more flexibility, a delta file can take the form of a python module. These should
be named `*.py`. Note that database-engine-specific modules are not supported here –
instead you can write `if isinstance(database_engine, PostgresEngine)` or similar.
A Python delta module should define either or both of the following functions:
```python
import synapse.config.homeserver
import synapse.storage.engines
import synapse.storage.types
def run_create(
cur: synapse.storage.types.Cursor,
database_engine: synapse.storage.engines.BaseDatabaseEngine,
) -> None:
"""Called whenever an existing or new database is to be upgraded"""
...
def run_upgrade(
cur: synapse.storage.types.Cursor,
database_engine: synapse.storage.engines.BaseDatabaseEngine,
config: synapse.config.homeserver.HomeServerConfig,
) -> None:
"""Called whenever an existing database is to be upgraded."""
...
```
## Boolean columns
Boolean columns require special treatment, since SQLite treats booleans the

7
docs/development/url_previews.md
View File

@@ -35,7 +35,12 @@ When Synapse is asked to preview a URL it does the following:
5. If the media is HTML:
1. Decodes the HTML via the stored file.
2. Generates an Open Graph response from the HTML.
3. If an image exists in the Open Graph response:
3. If a JSON oEmbed URL was found in the HTML via autodiscovery:
1. Downloads the URL and stores it into a file via the media storage provider
and saves the local media metadata.
2. Convert the oEmbed response to an Open Graph response.
3. Override any Open Graph data from the HTML with data from oEmbed.
4. If an image exists in the Open Graph response:
1. Downloads the URL and stores it into a file via the media storage
provider and saves the local media metadata.
2. Generates thumbnails.

62
docs/modules/password_auth_provider_callbacks.md
View File

@@ -105,6 +105,68 @@ device ID), and the (now deactivated) access token.
If multiple modules implement this callback, Synapse runs them all in order.
### `get_username_for_registration`
_First introduced in Synapse v1.52.0_
```python
async def get_username_for_registration(
uia_results: Dict[str, Any],
params: Dict[str, Any],
) -> Optional[str]
```
Called when registering a new user. The module can return a username to set for the user
being registered by returning it as a string, or `None` if it doesn't wish to force a
username for this user. If a username is returned, it will be used as the local part of a
user's full Matrix ID (e.g. it's `alice` in `@alice:example.com`).
This callback is called once [User-Interactive Authentication](https://spec.matrix.org/latest/client-server-api/#user-interactive-authentication-api)
has been completed by the user. It is not called when registering a user via SSO. It is
passed two dictionaries, which include the information that the user has provided during
the registration process.
The first dictionary contains the results of the [User-Interactive Authentication](https://spec.matrix.org/latest/client-server-api/#user-interactive-authentication-api)
flow followed by the user. Its keys are the identifiers of every step involved in the flow,
associated with either a boolean value indicating whether the step was correctly completed,
or additional information (e.g. email address, phone number...). A list of most existing
identifiers can be found in the [Matrix specification](https://spec.matrix.org/v1.1/client-server-api/#authentication-types).
Here's an example featuring all currently supported keys:
```python
{
"m.login.dummy": True, # Dummy authentication
"m.login.terms": True, # User has accepted the terms of service for the homeserver
"m.login.recaptcha": True, # User has completed the recaptcha challenge
"m.login.email.identity": { # User has provided and verified an email address
"medium": "email",
"address": "alice@example.com",
"validated_at": 1642701357084,
},
"m.login.msisdn": { # User has provided and verified a phone number
"medium": "msisdn",
"address": "33123456789",
"validated_at": 1642701357084,
},
"m.login.registration_token": "sometoken", # User has registered through a registration token
}
```
The second dictionary contains the parameters provided by the user's client in the request
to `/_matrix/client/v3/register`. See the [Matrix specification](https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3register)
for a complete list of these parameters.
If the module cannot, or does not wish to, generate a username for this user, it must
return `None`.
If multiple modules implement this callback, they will be considered in order. If a
callback returns `None`, Synapse falls through to the next one. The value of the first
callback that does not return `None` will be used. If this happens, Synapse will not call
any of the subsequent implementations of this callback. If every callback return `None`,
the username provided by the user is used, if any (otherwise one is automatically
generated).
## Example
The example module below implements authentication checkers for two different login types:

25
docs/openid.md
View File

@@ -390,9 +390,6 @@ oidc_providers:
### Facebook
Like Github, Facebook provide a custom OAuth2 API rather than an OIDC-compliant
one so requires a little more configuration.
0. You will need a Facebook developer account. You can register for one
[here](https://developers.facebook.com/async/registration/).
1. On the [apps](https://developers.facebook.com/apps/) page of the developer
@@ -412,24 +409,28 @@ Synapse config:
idp_name: Facebook
idp_brand: "facebook" # optional: styling hint for clients
discover: false
issuer: "https://facebook.com"
issuer: "https://www.facebook.com"
client_id: "your-client-id" # TO BE FILLED
client_secret: "your-client-secret" # TO BE FILLED
scopes: ["openid", "email"]
authorization_endpoint: https://facebook.com/dialog/oauth
token_endpoint: https://graph.facebook.com/v9.0/oauth/access_token
user_profile_method: "userinfo_endpoint"
userinfo_endpoint: "https://graph.facebook.com/v9.0/me?fields=id,name,email,picture"
authorization_endpoint: "https://facebook.com/dialog/oauth"
token_endpoint: "https://graph.facebook.com/v9.0/oauth/access_token"
jwks_uri: "https://www.facebook.com/.well-known/oauth/openid/jwks/"
user_mapping_provider:
config:
subject_claim: "id"
display_name_template: "{{ user.name }}"
email_template: "{{ '{{ user.email }}' }}"
```
Relevant documents:
* https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow
* Using Facebook's Graph API: https://developers.facebook.com/docs/graph-api/using-graph-api/
* Reference to the User endpoint: https://developers.facebook.com/docs/graph-api/reference/user
* [Manually Build a Login Flow](https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow)
* [Using Facebook's Graph API](https://developers.facebook.com/docs/graph-api/using-graph-api/)
* [Reference to the User endpoint](https://developers.facebook.com/docs/graph-api/reference/user)
Facebook do have an [OIDC discovery endpoint](https://www.facebook.com/.well-known/openid-configuration),
but it has a `response_types_supported` which excludes "code" (which we rely on, and
is even mentioned in their [documentation](https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow#login)),
so we have to disable discovery and configure the URIs manually.
### Gitea

81
docs/sample_config.yaml
View File

@@ -41,11 +41,11 @@
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}
#- module: my_super_module.MySuperClass
# config:
# do_thing: true
#- module: my_other_super_module.SomeClass
# config: {}
## Server ##
@@ -74,13 +74,7 @@ server_name: "SERVERNAME"
#
pid_file: DATADIR/homeserver.pid
# The absolute URL to the web client which /_matrix/client will redirect
# to if 'webclient' is configured under the 'listeners' configuration.
#
# This option can be also set to the filesystem path to the web client
# which will be served at /_matrix/client/ if 'webclient' is configured
# under the 'listeners' configuration, however this is a security risk:
# https://github.com/matrix-org/synapse#security-note
# The absolute URL to the web client which / will redirect to.
#
#web_client_location: https://riot.example.com/
@@ -164,7 +158,7 @@ presence:
# The default room version for newly created rooms.
#
# Known room versions are listed here:
# https://matrix.org/docs/spec/#complete-list-of-room-versions
# https://spec.matrix.org/latest/rooms/#complete-list-of-room-versions
#
# For example, for room version 1, default_room_version should be set
# to "1".
@@ -310,8 +304,6 @@ presence:
# static: static resources under synapse/static (/_matrix/static). (Mostly
# useful for 'fallback authentication'.)
#
# webclient: A web client. Requires web_client_location to be set.
#
listeners:
# TLS-enabled listener: for when matrix traffic is sent directly to synapse.
#
@@ -479,6 +471,20 @@ limit_remote_rooms:
#
#allow_per_room_profiles: false
# The largest allowed file size for a user avatar. Defaults to no restriction.
#
# Note that user avatar changes will not work if this is set without
# using Synapse's media repository.
#
#max_avatar_size: 10M
# The MIME types allowed for user avatars. Defaults to no restriction.
#
# Note that user avatar changes will not work if this is set without
# using Synapse's media repository.
#
#allowed_avatar_mimetypes: ["image/png", "image/jpeg", "image/gif"]
# How long to keep redacted events in unredacted form in the database. After
# this period redacted events get replaced with their redacted form in the DB.
#
@@ -851,6 +857,9 @@ log_config: "CONFDIR/SERVERNAME.log.config"
# - one for ratelimiting how often a user or IP can attempt to validate a 3PID.
# - two for ratelimiting how often invites can be sent in a room or to a
# specific user.
# - one for ratelimiting 3PID invites (i.e. invites sent to a third-party ID
# such as an email address or a phone number) based on the account that's
# sending the invite.
#
# The defaults are as shown below.
#
@@ -900,6 +909,10 @@ log_config: "CONFDIR/SERVERNAME.log.config"
# per_user:
# per_second: 0.003
# burst_count: 5
#
#rc_third_party_invite:
# per_second: 0.2
# burst_count: 10
# Ratelimiting settings for incoming federation
#
@@ -1436,6 +1449,16 @@ account_threepid_delegates:
#
#auto_join_rooms_for_guests: false
# Whether to inhibit errors raised when registering a new account if the user ID
# already exists. If turned on, that requests to /register/available will always
# show a user ID as available, and Synapse won't raise an error when starting
# a registration with a user ID that already exists. However, Synapse will still
# raise an error if the registration completes and the username conflicts.
#
# Defaults to false.
#
#inhibit_user_in_use_error: true
## Metrics ###
@@ -1503,11 +1526,20 @@ room_prejoin_state:
#additional_event_types:
# - org.example.custom.event.type
# If enabled, puppeted user IP's can also be tracked. By default when
# puppeting another user, the user who has created the access token
# for puppeting is tracked. If this is enabled, both requests are tracked.
# Implicitly enables MAU tracking for puppeted users.
#track_puppeted_user_ips: false
# We record the IP address of clients used to access the API for various
# reasons, including displaying it to the user in the "Where you're signed in"
# dialog.
#
# By default, when puppeting another user via the admin API, the client IP
# address is recorded against the user who created the access token (ie, the
# admin user), and *not* the puppeted user.
#
# Uncomment the following to also record the IP address against the puppeted
# user. (This also means that the puppeted user will count as an "active" user
# for the purpose of monthly active user tracking - see 'limit_usage_by_mau' etc
# above.)
#
#track_puppeted_user_ips: true
# A list of application service config files to use
@@ -1876,10 +1908,13 @@ saml2_config:
# Defaults to false. Avoid this in production.
#
# user_profile_method: Whether to fetch the user profile from the userinfo
# endpoint. Valid values are: 'auto' or 'userinfo_endpoint'.
# endpoint, or to rely on the data returned in the id_token from the
# token_endpoint.
#
# Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is
# included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the
# Valid values are: 'auto' or 'userinfo_endpoint'.
#
# Defaults to 'auto', which uses the userinfo endpoint if 'openid' is
# not included in 'scopes'. Set to 'userinfo_endpoint' to always use the
# userinfo endpoint.
#
# allow_existing_users: set to 'true' to allow a user logging in via OIDC to

2
docs/setup/installation.md
View File

@@ -194,7 +194,7 @@ When following this route please make sure that the [Platform-specific prerequis
System requirements:
- POSIX-compliant system (tested on Linux & OS X)
- Python 3.6 or later, up to Python 3.9.
- Python 3.7 or later, up to Python 3.10.
- At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
To install the Synapse homeserver run:

16
docs/turn-howto.md
View File

@@ -137,6 +137,10 @@ This will install and start a systemd service called `coturn`.
# TLS private key file
pkey=/path/to/privkey.pem
# Ensure the configuration lines that disable TLS/DTLS are commented-out or removed
#no-tls
#no-dtls
```
In this case, replace the `turn:` schemes in the `turn_uris` settings below
@@ -145,6 +149,14 @@ This will install and start a systemd service called `coturn`.
We recommend that you only try to set up TLS/DTLS once you have set up a
basic installation and got it working.
NB: If your TLS certificate was provided by Let's Encrypt, TLS/DTLS will
not work with any Matrix client that uses Chromium's WebRTC library. This
currently includes Element Android & iOS; for more details, see their
[respective](https://github.com/vector-im/element-android/issues/1533)
[issues](https://github.com/vector-im/element-ios/issues/2712) as well as the underlying
[WebRTC issue](https://bugs.chromium.org/p/webrtc/issues/detail?id=11710).
Consider using a ZeroSSL certificate for your TURN server as a working alternative.
1. Ensure your firewall allows traffic into the TURN server on the ports
you've configured it to listen on (By default: 3478 and 5349 for TURN
traffic (remember to allow both TCP and UDP traffic), and ports 49152-65535
@@ -250,6 +262,10 @@ Here are a few things to try:
* Check that you have opened your firewall to allow UDP traffic to the UDP
relay ports (49152-65535 by default).
* Try disabling `coturn`'s TLS/DTLS listeners and enable only its (unencrypted)
TCP/UDP listeners. (This will only leave signaling traffic unencrypted;
voice & video WebRTC traffic is always encrypted.)
* Some WebRTC implementations (notably, that of Google Chrome) appear to get
confused by TURN servers which are reachable over IPv6 (this appears to be
an unexpected side-effect of its handling of multiple IP addresses as

45
docs/upgrade.md
View File

@@ -84,6 +84,45 @@ process, for example:
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
```
# Upgrading to v1.(next)
## Stablisation of MSC3231
The unstable validity-check endpoint for the
[Registration Tokens](https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv1registermloginregistration_tokenvalidity)
feature has been stabilised and moved from:
`/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity`
to:
`/_matrix/client/v1/register/m.login.registration_token/validity`
Please update any relevant reverse proxy or firewall configurations appropriately.
# Upgrading to v1.53.0
## Dropping support for `webclient` listeners and non-HTTP(S) `web_client_location`
Per the deprecation notice in Synapse v1.51.0, listeners of type `webclient`
are no longer supported and configuring them is a now a configuration error.
Configuring a non-HTTP(S) `web_client_location` configuration is is now a
configuration error. Since the `webclient` listener is no longer supported, this
setting only applies to the root path `/` of Synapse's web server and no longer
the `/_matrix/client/` path.
# Upgrading to v1.51.0
## Deprecation of `webclient` listeners and non-HTTP(S) `web_client_location`
Listeners of type `webclient` are deprecated and scheduled to be removed in
Synapse v1.53.0.
Similarly, a non-HTTP(S) `web_client_location` configuration is deprecated and
will become a configuration error in Synapse v1.53.0.
# Upgrading to v1.50.0
@@ -1118,8 +1157,7 @@ more details on upgrading your database.
Synapse v1.0 is the first release to enforce validation of TLS
certificates for the federation API. It is therefore essential that your
certificates are correctly configured. See the
[FAQ](MSC1711_certificates_FAQ.md) for more information.
certificates are correctly configured.
Note, v1.0 installations will also no longer be able to federate with
servers that have not correctly configured their certificates.
@@ -1184,9 +1222,6 @@ you will need to replace any self-signed certificates with those
verified by a root CA. Information on how to do so can be found at the
ACME docs.
For more information on configuring TLS certificates see the
[FAQ](MSC1711_certificates_FAQ.md).
# Upgrading to v0.34.0
1. This release is the first to fully support Python 3. Synapse will

100
docs/usage/administration/admin_api/federation.md
View File

@@ -86,7 +86,7 @@ The following fields are returned in the JSON response body:
- `next_token`: string representing a positive integer - Indication for pagination. See above.
- `total` - integer - Total number of destinations.
# Destination Details API
## Destination Details API
This API gets the retry timing info for a specific remote server.
@@ -108,7 +108,105 @@ A response body like the following is returned:
}
```
**Parameters**
The following parameters should be set in the URL:
- `destination` - Name of the remote server.
**Response**
The response fields are the same like in the `destinations` array in
[List of destinations](#list-of-destinations) response.
## Destination rooms
This API gets the rooms that federate with a specific remote server.
The API is:
```
GET /_synapse/admin/v1/federation/destinations/<destination>/rooms
```
A response body like the following is returned:
```json
{
"rooms":[
{
"room_id": "!OGEhHVWSdvArJzumhm:matrix.org",
"stream_ordering": 8326
},
{
"room_id": "!xYvNcQPhnkrdUmYczI:matrix.org",
"stream_ordering": 93534
}
],
"total": 2
}
```
To paginate, check for `next_token` and if present, call the endpoint again
with `from` set to the value of `next_token`. This will return a new page.
If the endpoint does not return a `next_token` then there are no more destinations
to paginate through.
**Parameters**
The following parameters should be set in the URL:
- `destination` - Name of the remote server.
The following query parameters are available:
- `from` - Offset in the returned list. Defaults to `0`.
- `limit` - Maximum amount of destinations to return. Defaults to `100`.
- `dir` - Direction of room order by `room_id`. Either `f` for forwards or `b` for
backwards. Defaults to `f`.
**Response**
The following fields are returned in the JSON response body:
- `rooms` - An array of objects, each containing information about a room.
Room objects contain the following fields:
- `room_id` - string - The ID of the room.
- `stream_ordering` - integer - The stream ordering of the most recent
successfully-sent [PDU](understanding_synapse_through_grafana_graphs.md#federation)
to this destination in this room.
- `next_token`: string representing a positive integer - Indication for pagination. See above.
- `total` - integer - Total number of destinations.
## Reset connection timeout
Synapse makes federation requests to other homeservers. If a federation request fails,
Synapse will mark the destination homeserver as offline, preventing any future requests
to that server for a "cooldown" period. This period grows over time if the server
continues to fail its responses
([exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff)).
Admins can cancel the cooldown period with this API.
This API resets the retry timing for a specific remote server and tries to connect to
the remote server again. It does not wait for the next `retry_interval`.
The connection must have previously run into an error and `retry_last_ts`
([Destination Details API](#destination-details-api)) must not be equal to `0`.
The connection attempt is carried out in the background and can take a while
even if the API already returns the http status 200.
The API is:
```
POST /_synapse/admin/v1/federation/destinations/<destination>/reset_connection
{}
```
**Parameters**
The following parameters should be set in the URL:
- `destination` - Name of the remote server.

2
docs/workers.md
View File

@@ -241,7 +241,7 @@ expressions:
# Registration/login requests
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
^/_matrix/client/(r0|v3|unstable)/register$
^/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity$
^/_matrix/client/v1/register/m.login.registration_token/validity$
# Event sending requests
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact

3
mypy.ini
View File

@@ -77,9 +77,6 @@ exclude = (?x)
|tests/push/test_http.py
|tests/push/test_presentable_names.py
|tests/push/test_push_rule_evaluator.py
|tests/rest/admin/test_admin.py
|tests/rest/admin/test_user.py
|tests/rest/admin/test_username_available.py
|tests/rest/client/test_account.py
|tests/rest/client/test_events.py
|tests/rest/client/test_filter.py

14
scripts-dev/complement.sh
View File

@@ -8,7 +8,8 @@
# By default the script will fetch the latest Complement master branch and
# run tests with that. This can be overridden to use a custom Complement
# checkout by setting the COMPLEMENT_DIR environment variable to the
# filepath of a local Complement checkout.
# filepath of a local Complement checkout or by setting the COMPLEMENT_REF
# environment variable to pull a different branch or commit.
#
# By default Synapse is run in monolith mode. This can be overridden by
# setting the WORKERS environment variable.
@@ -31,11 +32,12 @@ cd "$(dirname $0)/.."
# Check for a user-specified Complement checkout
if [[ -z "$COMPLEMENT_DIR" ]]; then
echo "COMPLEMENT_DIR not set. Fetching the latest Complement checkout..."
wget -Nq https://github.com/matrix-org/complement/archive/master.tar.gz
tar -xzf master.tar.gz
COMPLEMENT_DIR=complement-master
echo "Checkout available at 'complement-master'"
COMPLEMENT_REF=${COMPLEMENT_REF:-master}
echo "COMPLEMENT_DIR not set. Fetching Complement checkout from ${COMPLEMENT_REF}..."
wget -Nq https://github.com/matrix-org/complement/archive/${COMPLEMENT_REF}.tar.gz
tar -xzf ${COMPLEMENT_REF}.tar.gz
COMPLEMENT_DIR=complement-${COMPLEMENT_REF}
echo "Checkout available at 'complement-${COMPLEMENT_REF}'"
fi
# Build the base Synapse image from the local checkout

4
scripts/synapse_port_db
View File

@@ -36,6 +36,8 @@ from synapse.logging.context import (
run_in_background,
)
from synapse.storage.database import DatabasePool, make_conn
from synapse.storage.databases.main import PushRuleStore
from synapse.storage.databases.main.account_data import AccountDataWorkerStore
from synapse.storage.databases.main.client_ips import ClientIpBackgroundUpdateStore
from synapse.storage.databases.main.deviceinbox import DeviceInboxBackgroundUpdateStore
from synapse.storage.databases.main.devices import DeviceBackgroundUpdateStore
@@ -180,6 +182,8 @@ class Store(
UserDirectoryBackgroundUpdateStore,
EndToEndKeyBackgroundStore,
StatsStore,
AccountDataWorkerStore,
PushRuleStore,
PusherWorkerStore,
PresenceBackgroundUpdateStore,
GroupServerWorkerStore,

2
setup.py
View File

@@ -150,7 +150,7 @@ setup(
zip_safe=False,
long_description=long_description,
long_description_content_type="text/x-rst",
python_requires="~=3.6",
python_requires="~=3.7",
entry_points={
"console_scripts": [
"synapse_homeserver = synapse.app.homeserver:main",

6
synapse/__init__.py
View File

@@ -21,8 +21,8 @@ import os
import sys
# Check that we're not running on an unsupported Python version.
if sys.version_info < (3, 6):
print("Synapse requires Python 3.6 or above.")
if sys.version_info < (3, 7):
print("Synapse requires Python 3.7 or above.")
sys.exit(1)
# Twisted and canonicaljson will fail to import when this file is executed to
@@ -47,7 +47,7 @@ try:
except ImportError:
pass
__version__ = "1.50.0rc1"
__version__ = "1.52.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

18
synapse/_scripts/review_recent_signups.py
View File

@@ -46,7 +46,9 @@ class UserInfo:
ips: List[str] = attr.Factory(list)
def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
def get_recent_users(
txn: LoggingTransaction, since_ms: int, exclude_app_service: bool
) -> List[UserInfo]:
"""Fetches recently registered users and some info on them."""
sql = """
@@ -56,6 +58,9 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
AND deactivated = 0
"""
if exclude_app_service:
sql += " AND appservice_id IS NULL"
txn.execute(sql, (since_ms / 1000,))
user_infos = [UserInfo(user_id, creation_ts) for user_id, creation_ts in txn]
@@ -113,7 +118,7 @@ def main() -> None:
"-e",
"--exclude-emails",
action="store_true",
help="Exclude users that have validated email addresses",
help="Exclude users that have validated email addresses.",
)
parser.add_argument(
"-u",
@@ -121,6 +126,12 @@ def main() -> None:
action="store_true",
help="Only print user IDs that match.",
)
parser.add_argument(
"-a",
"--exclude-app-service",
help="Exclude appservice users.",
action="store_true",
)
config = ReviewConfig()
@@ -133,6 +144,7 @@ def main() -> None:
since_ms = time.time() * 1000 - Config.parse_duration(config_args.since)
exclude_users_with_email = config_args.exclude_emails
exclude_users_with_appservice = config_args.exclude_app_service
include_context = not config_args.only_users
for database_config in config.database.databases:
@@ -143,7 +155,7 @@ def main() -> None:
with make_conn(database_config, engine, "review_recent_signups") as db_conn:
# This generates a type of Cursor, not LoggingTransaction.
user_infos = get_recent_users(db_conn.cursor(), since_ms) # type: ignore[arg-type]
user_infos = get_recent_users(db_conn.cursor(), since_ms, exclude_users_with_appservice) # type: ignore[arg-type]
for user_info in user_infos:
if exclude_users_with_email and user_info.emails:

2
synapse/api/constants.py
View File

@@ -81,7 +81,7 @@ class LoginType:
TERMS: Final = "m.login.terms"
SSO: Final = "m.login.sso"
DUMMY: Final = "m.login.dummy"
REGISTRATION_TOKEN: Final = "org.matrix.msc3231.login.registration_token"
REGISTRATION_TOKEN: Final = "m.login.registration_token"
# This is used in the `type` parameter for /register when called by

30
synapse/api/room_versions.py
View File

@@ -46,41 +46,41 @@ class RoomDisposition:
UNSTABLE = "unstable"
@attr.s(slots=True, frozen=True)
@attr.s(slots=True, frozen=True, auto_attribs=True)
class RoomVersion:
"""An object which describes the unique attributes of a room version."""
identifier = attr.ib(type=str) # the identifier for this version
disposition = attr.ib(type=str) # one of the RoomDispositions
event_format = attr.ib(type=int) # one of the EventFormatVersions
state_res = attr.ib(type=int) # one of the StateResolutionVersions
enforce_key_validity = attr.ib(type=bool)
identifier: str # the identifier for this version
disposition: str # one of the RoomDispositions
event_format: int # one of the EventFormatVersions
state_res: int # one of the StateResolutionVersions
enforce_key_validity: bool
# Before MSC2432, m.room.aliases had special auth rules and redaction rules
special_case_aliases_auth = attr.ib(type=bool)
special_case_aliases_auth: bool
# Strictly enforce canonicaljson, do not allow:
# * Integers outside the range of [-2 ^ 53 + 1, 2 ^ 53 - 1]
# * Floats
# * NaN, Infinity, -Infinity
strict_canonicaljson = attr.ib(type=bool)
strict_canonicaljson: bool
# MSC2209: Check 'notifications' key while verifying
# m.room.power_levels auth rules.
limit_notifications_power_levels = attr.ib(type=bool)
limit_notifications_power_levels: bool
# MSC2174/MSC2176: Apply updated redaction rules algorithm.
msc2176_redaction_rules = attr.ib(type=bool)
msc2176_redaction_rules: bool
# MSC3083: Support the 'restricted' join_rule.
msc3083_join_rules = attr.ib(type=bool)
msc3083_join_rules: bool
# MSC3375: Support for the proper redaction rules for MSC3083. This mustn't
# be enabled if MSC3083 is not.
msc3375_redaction_rules = attr.ib(type=bool)
msc3375_redaction_rules: bool
# MSC2403: Allows join_rules to be set to 'knock', changes auth rules to allow sending
# m.room.membership event with membership 'knock'.
msc2403_knocking = attr.ib(type=bool)
msc2403_knocking: bool
# MSC2716: Adds m.room.power_levels -> content.historical field to control
# whether "insertion", "chunk", "marker" events can be sent
msc2716_historical = attr.ib(type=bool)
msc2716_historical: bool
# MSC2716: Adds support for redacting "insertion", "chunk", and "marker" events
msc2716_redactions = attr.ib(type=bool)
msc2716_redactions: bool
class RoomVersions:

1
synapse/api/urls.py
View File

@@ -28,7 +28,6 @@ FEDERATION_V1_PREFIX = FEDERATION_PREFIX + "/v1"
FEDERATION_V2_PREFIX = FEDERATION_PREFIX + "/v2"
FEDERATION_UNSTABLE_PREFIX = FEDERATION_PREFIX + "/unstable"
STATIC_PREFIX = "/_matrix/static"
WEB_CLIENT_PREFIX = "/_matrix/client"
SERVER_KEY_V2_PREFIX = "/_matrix/key/v2"
MEDIA_R0_PREFIX = "/_matrix/media/r0"
MEDIA_V3_PREFIX = "/_matrix/media/v3"

17
synapse/app/_base.py
View File

@@ -16,7 +16,6 @@ import atexit
import gc
import logging
import os
import platform
import signal
import socket
import sys
@@ -60,7 +59,7 @@ from synapse.events.spamcheck import load_legacy_spam_checkers
from synapse.events.third_party_rules import load_legacy_third_party_event_rules
from synapse.handlers.auth import load_legacy_password_auth_providers
from synapse.logging.context import PreserveLoggingContext
from synapse.metrics import register_threadpool
from synapse.metrics import install_gc_manager, register_threadpool
from synapse.metrics.background_process_metrics import wrap_as_background_process
from synapse.metrics.jemalloc import setup_jemalloc_stats
from synapse.types import ISynapseReactor
@@ -159,6 +158,7 @@ def start_reactor(
change_resource_limit(soft_file_limit)
if gc_thresholds:
gc.set_threshold(*gc_thresholds)
install_gc_manager()
run_command()
# make sure that we run the reactor with the sentinel log context,
@@ -435,7 +435,8 @@ async def start(hs: "HomeServer") -> None:
# before we start the listeners.
module_api = hs.get_module_api()
for module, config in hs.config.modules.loaded_modules:
module(config=config, api=module_api)
m = module(config=config, api=module_api)
logger.info("Loaded module %s", m)
load_legacy_spam_checkers(hs)
load_legacy_third_party_event_rules(hs)
@@ -467,15 +468,13 @@ async def start(hs: "HomeServer") -> None:
# everything currently allocated are things that will be used for the
# rest of time. Doing so means less work each GC (hopefully).
#
# This only works on Python 3.7
if platform.python_implementation() == "CPython" and sys.version_info >= (3, 7):
# PyPy does not (yet?) implement gc.freeze()
if hasattr(gc, "freeze"):
gc.collect()
gc.freeze()
# Speed up shutdowns by freezing all allocated objects. This moves everything
# into the permanent generation and excludes them from the final GC.
# Unfortunately only works on Python 3.7
if platform.python_implementation() == "CPython" and sys.version_info >= (3, 7):
# Speed up shutdowns by freezing all allocated objects. This moves everything
# into the permanent generation and excludes them from the final GC.
atexit.register(gc.freeze)

46
synapse/app/homeserver.py
View File

@@ -21,20 +21,17 @@ from typing import Dict, Iterable, Iterator, List
from twisted.internet.tcp import Port
from twisted.web.resource import EncodingResourceWrapper, Resource
from twisted.web.server import GzipEncoderFactory
from twisted.web.static import File
import synapse
import synapse.config.logger
from synapse import events
from synapse.api.urls import (
CLIENT_API_PREFIX,
FEDERATION_PREFIX,
LEGACY_MEDIA_PREFIX,
MEDIA_R0_PREFIX,
MEDIA_V3_PREFIX,
SERVER_KEY_V2_PREFIX,
STATIC_PREFIX,
WEB_CLIENT_PREFIX,
)
from synapse.app import _base
from synapse.app._base import (
@@ -54,7 +51,6 @@ from synapse.http.additional_resource import AdditionalResource
from synapse.http.server import (
OptionsResource,
RootOptionsRedirectResource,
RootRedirect,
StaticResource,
)
from synapse.http.site import SynapseSite
@@ -132,9 +128,15 @@ class SynapseHomeServer(HomeServer):
resources.update(self._module_web_resources)
self._module_web_resources_consumed = True
# try to find something useful to redirect '/' to
if WEB_CLIENT_PREFIX in resources:
root_resource: Resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
# Try to find something useful to serve at '/':
#
# 1. Redirect to the web client if it is an HTTP(S) URL.
# 2. Redirect to the static "Synapse is running" page.
# 3. Do not redirect and use a blank resource.
if self.config.server.web_client_location:
root_resource: Resource = RootOptionsRedirectResource(
self.config.server.web_client_location
)
elif STATIC_PREFIX in resources:
root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
else:
@@ -193,7 +195,13 @@ class SynapseHomeServer(HomeServer):
resources.update(
{
CLIENT_API_PREFIX: client_resource,
"/_matrix/client/api/v1": client_resource,
"/_matrix/client/r0": client_resource,
"/_matrix/client/v1": client_resource,
"/_matrix/client/v3": client_resource,
"/_matrix/client/unstable": client_resource,
"/_matrix/client/v2_alpha": client_resource,
"/_matrix/client/versions": client_resource,
"/.well-known": well_known_resource(self),
"/_synapse/admin": AdminRestResource(self),
**build_synapse_client_resource_tree(self),
@@ -256,28 +264,6 @@ class SynapseHomeServer(HomeServer):
if name in ["keys", "federation"]:
resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)
if name == "webclient":
webclient_loc = self.config.server.web_client_location
if webclient_loc is None:
logger.warning(
"Not enabling webclient resource, as web_client_location is unset."
)
elif webclient_loc.startswith("http://") or webclient_loc.startswith(
"https://"
):
resources[WEB_CLIENT_PREFIX] = RootRedirect(webclient_loc)
else:
logger.warning(
"Running webclient on the same domain is not recommended: "
"https://github.com/matrix-org/synapse#security-note - "
"after you move webclient to different host you can set "
"web_client_location to its full URL to enable redirection."
)
# GZip is disabled here due to
# https://twistedmatrix.com/trac/ticket/7678
resources[WEB_CLIENT_PREFIX] = File(webclient_loc)
if name == "metrics" and self.config.metrics.enable_metrics:
resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)

3
synapse/appservice/__init__.py
View File

@@ -351,11 +351,13 @@ class AppServiceTransaction:
id: int,
events: List[EventBase],
ephemeral: List[JsonDict],
to_device_messages: List[JsonDict],
):
self.service = service
self.id = id
self.events = events
self.ephemeral = ephemeral
self.to_device_messages = to_device_messages
async def send(self, as_api: "ApplicationServiceApi") -> bool:
"""Sends this transaction using the provided AS API interface.
@@ -369,6 +371,7 @@ class AppServiceTransaction:
service=self.service,
events=self.events,
ephemeral=self.ephemeral,
to_device_messages=self.to_device_messages,
txn_id=self.id,
)

29
synapse/appservice/api.py
View File

@@ -218,8 +218,23 @@ class ApplicationServiceApi(SimpleHttpClient):
service: "ApplicationService",
events: List[EventBase],
ephemeral: List[JsonDict],
to_device_messages: List[JsonDict],
txn_id: Optional[int] = None,
) -> bool:
"""
Push data to an application service.
Args:
service: The application service to send to.
events: The persistent events to send.
ephemeral: The ephemeral events to send.
to_device_messages: The to-device messages to send.
txn_id: An unique ID to assign to this transaction. Application services should
deduplicate transactions received with identitical IDs.
Returns:
True if the task succeeded, False if it failed.
"""
if service.url is None:
return True
@@ -237,13 +252,15 @@ class ApplicationServiceApi(SimpleHttpClient):
uri = service.url + ("/transactions/%s" % urllib.parse.quote(str(txn_id)))
# Never send ephemeral events to appservices that do not support it
body: Dict[str, List[JsonDict]] = {"events": serialized_events}
if service.supports_ephemeral:
body = {
"events": serialized_events,
"de.sorunome.msc2409.ephemeral": ephemeral,
}
else:
body = {"events": serialized_events}
body.update(
{
# TODO: Update to stable prefixes once MSC2409 completes FCP merge.
"de.sorunome.msc2409.ephemeral": ephemeral,
"de.sorunome.msc2409.to_device": to_device_messages,
}
)
try:
await self.put_json(

97
synapse/appservice/scheduler.py
View File

@@ -48,7 +48,16 @@ This is all tied together by the AppServiceScheduler which DIs the required
components.
"""
import logging
from typing import TYPE_CHECKING, Awaitable, Callable, Dict, List, Optional, Set
from typing import (
TYPE_CHECKING,
Awaitable,
Callable,
Collection,
Dict,
List,
Optional,
Set,
)
from synapse.appservice import ApplicationService, ApplicationServiceState
from synapse.appservice.api import ApplicationServiceApi
@@ -71,6 +80,9 @@ MAX_PERSISTENT_EVENTS_PER_TRANSACTION = 100
# Maximum number of ephemeral events to provide in an AS transaction.
MAX_EPHEMERAL_EVENTS_PER_TRANSACTION = 100
# Maximum number of to-device messages to provide in an AS transaction.
MAX_TO_DEVICE_MESSAGES_PER_TRANSACTION = 100
class ApplicationServiceScheduler:
"""Public facing API for this module. Does the required DI to tie the
@@ -97,15 +109,40 @@ class ApplicationServiceScheduler:
for service in services:
self.txn_ctrl.start_recoverer(service)
def submit_event_for_as(
self, service: ApplicationService, event: EventBase
def enqueue_for_appservice(
self,
appservice: ApplicationService,
events: Optional[Collection[EventBase]] = None,
ephemeral: Optional[Collection[JsonDict]] = None,
to_device_messages: Optional[Collection[JsonDict]] = None,
) -> None:
self.queuer.enqueue_event(service, event)
"""
Enqueue some data to be sent off to an application service.
def submit_ephemeral_events_for_as(
self, service: ApplicationService, events: List[JsonDict]
) -> None:
self.queuer.enqueue_ephemeral(service, events)
Args:
appservice: The application service to create and send a transaction to.
events: The persistent room events to send.
ephemeral: The ephemeral events to send.
to_device_messages: The to-device messages to send. These differ from normal
to-device messages sent to clients, as they have 'to_device_id' and
'to_user_id' fields.
"""
# We purposefully allow this method to run with empty events/ephemeral
# collections, so that callers do not need to check iterable size themselves.
if not events and not ephemeral and not to_device_messages:
return
if events:
self.queuer.queued_events.setdefault(appservice.id, []).extend(events)
if ephemeral:
self.queuer.queued_ephemeral.setdefault(appservice.id, []).extend(ephemeral)
if to_device_messages:
self.queuer.queued_to_device_messages.setdefault(appservice.id, []).extend(
to_device_messages
)
# Kick off a new application service transaction
self.queuer.start_background_request(appservice)
class _ServiceQueuer:
@@ -121,13 +158,15 @@ class _ServiceQueuer:
self.queued_events: Dict[str, List[EventBase]] = {}
# dict of {service_id: [events]}
self.queued_ephemeral: Dict[str, List[JsonDict]] = {}
# dict of {service_id: [to_device_message_json]}
self.queued_to_device_messages: Dict[str, List[JsonDict]] = {}
# the appservices which currently have a transaction in flight
self.requests_in_flight: Set[str] = set()
self.txn_ctrl = txn_ctrl
self.clock = clock
def _start_background_request(self, service: ApplicationService) -> None:
def start_background_request(self, service: ApplicationService) -> None:
# start a sender for this appservice if we don't already have one
if service.id in self.requests_in_flight:
return
@@ -136,16 +175,6 @@ class _ServiceQueuer:
"as-sender-%s" % (service.id,), self._send_request, service
)
def enqueue_event(self, service: ApplicationService, event: EventBase) -> None:
self.queued_events.setdefault(service.id, []).append(event)
self._start_background_request(service)
def enqueue_ephemeral(
self, service: ApplicationService, events: List[JsonDict]
) -> None:
self.queued_ephemeral.setdefault(service.id, []).extend(events)
self._start_background_request(service)
async def _send_request(self, service: ApplicationService) -> None:
# sanity-check: we shouldn't get here if this service already has a sender
# running.
@@ -162,11 +191,21 @@ class _ServiceQueuer:
ephemeral = all_events_ephemeral[:MAX_EPHEMERAL_EVENTS_PER_TRANSACTION]
del all_events_ephemeral[:MAX_EPHEMERAL_EVENTS_PER_TRANSACTION]
if not events and not ephemeral:
all_to_device_messages = self.queued_to_device_messages.get(
service.id, []
)
to_device_messages_to_send = all_to_device_messages[
:MAX_TO_DEVICE_MESSAGES_PER_TRANSACTION
]
del all_to_device_messages[:MAX_TO_DEVICE_MESSAGES_PER_TRANSACTION]
if not events and not ephemeral and not to_device_messages_to_send:
return
try:
await self.txn_ctrl.send(service, events, ephemeral)
await self.txn_ctrl.send(
service, events, ephemeral, to_device_messages_to_send
)
except Exception:
logger.exception("AS request failed")
finally:
@@ -198,10 +237,24 @@ class _TransactionController:
service: ApplicationService,
events: List[EventBase],
ephemeral: Optional[List[JsonDict]] = None,
to_device_messages: Optional[List[JsonDict]] = None,
) -> None:
"""
Create a transaction with the given data and send to the provided
application service.
Args:
service: The application service to send the transaction to.
events: The persistent events to include in the transaction.
ephemeral: The ephemeral events to include in the transaction.
to_device_messages: The to-device messages to include in the transaction.
"""
try:
txn = await self.store.create_appservice_txn(
service=service, events=events, ephemeral=ephemeral or []
service=service,
events=events,
ephemeral=ephemeral or [],
to_device_messages=to_device_messages or [],
)
service_is_up = await self._is_service_up(service)
if service_is_up:

19
synapse/config/api.py
View File

@@ -61,11 +61,20 @@ class ApiConfig(Config):
#additional_event_types:
# - org.example.custom.event.type
# If enabled, puppeted user IP's can also be tracked. By default when
# puppeting another user, the user who has created the access token
# for puppeting is tracked. If this is enabled, both requests are tracked.
# Implicitly enables MAU tracking for puppeted users.
#track_puppeted_user_ips: false
# We record the IP address of clients used to access the API for various
# reasons, including displaying it to the user in the "Where you're signed in"
# dialog.
#
# By default, when puppeting another user via the admin API, the client IP
# address is recorded against the user who created the access token (ie, the
# admin user), and *not* the puppeted user.
#
# Uncomment the following to also record the IP address against the puppeted
# user. (This also means that the puppeted user will count as an "active" user
# for the purpose of monthly active user tracking - see 'limit_usage_by_mau' etc
# above.)
#
#track_puppeted_user_ips: true
""" % {
"formatted_default_state_types": formatted_default_state_types
}

24
synapse/config/emailconfig.py
View File

@@ -55,19 +55,19 @@ https://matrix-org.github.io/synapse/latest/templates.html
---------------------------------------------------------------------------------------"""
@attr.s(slots=True, frozen=True)
@attr.s(slots=True, frozen=True, auto_attribs=True)
class EmailSubjectConfig:
message_from_person_in_room = attr.ib(type=str)
message_from_person = attr.ib(type=str)
messages_from_person = attr.ib(type=str)
messages_in_room = attr.ib(type=str)
messages_in_room_and_others = attr.ib(type=str)
messages_from_person_and_others = attr.ib(type=str)
invite_from_person = attr.ib(type=str)
invite_from_person_to_room = attr.ib(type=str)
invite_from_person_to_space = attr.ib(type=str)
password_reset = attr.ib(type=str)
email_validation = attr.ib(type=str)
message_from_person_in_room: str
message_from_person: str
messages_from_person: str
messages_in_room: str
messages_in_room_and_others: str
messages_from_person_and_others: str
invite_from_person: str
invite_from_person_to_room: str
invite_from_person_to_space: str
password_reset: str
email_validation: str
class EmailConfig(Config):

9
synapse/config/experimental.py
View File

@@ -24,8 +24,6 @@ class ExperimentalConfig(Config):
def read_config(self, config: JsonDict, **kwargs):
experimental = config.get("experimental_features") or {}
# Whether to enable experimental MSC1849 (aka relations) support
self.msc1849_enabled = config.get("experimental_msc1849_support_enabled", True)
# MSC3440 (thread relation)
self.msc3440_enabled: bool = experimental.get("msc3440_enabled", False)
@@ -54,3 +52,10 @@ class ExperimentalConfig(Config):
self.msc3202_device_masquerading_enabled: bool = experimental.get(
"msc3202_device_masquerading", False
)
# MSC2409 (this setting only relates to optionally sending to-device messages).
# Presence, typing and read receipt EDUs are already sent to application services that
# have opted in to receive them. If enabled, this adds to-device messages to that list.
self.msc2409_to_device_messages_enabled: bool = experimental.get(
"msc2409_to_device_messages_enabled", False
)

10
synapse/config/modules.py
View File

@@ -41,9 +41,9 @@ class ModulesConfig(Config):
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}
#- module: my_super_module.MySuperClass
# config:
# do_thing: true
#- module: my_other_super_module.SomeClass
# config: {}
"""

9
synapse/config/oidc.py
View File

@@ -148,10 +148,13 @@ class OIDCConfig(Config):
# Defaults to false. Avoid this in production.
#
# user_profile_method: Whether to fetch the user profile from the userinfo
# endpoint. Valid values are: 'auto' or 'userinfo_endpoint'.
# endpoint, or to rely on the data returned in the id_token from the
# token_endpoint.
#
# Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is
# included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the
# Valid values are: 'auto' or 'userinfo_endpoint'.
#
# Defaults to 'auto', which uses the userinfo endpoint if 'openid' is
# not included in 'scopes'. Set to 'userinfo_endpoint' to always use the
# userinfo endpoint.
#
# allow_existing_users: set to 'true' to allow a user logging in via OIDC to

15
synapse/config/ratelimiting.py
View File

@@ -134,6 +134,14 @@ class RatelimitConfig(Config):
defaults={"per_second": 0.003, "burst_count": 5},
)
self.rc_third_party_invite = RateLimitConfig(
config.get("rc_third_party_invite", {}),
defaults={
"per_second": self.rc_message.per_second,
"burst_count": self.rc_message.burst_count,
},
)
def generate_config_section(self, **kwargs):
return """\
## Ratelimiting ##
@@ -168,6 +176,9 @@ class RatelimitConfig(Config):
# - one for ratelimiting how often a user or IP can attempt to validate a 3PID.
# - two for ratelimiting how often invites can be sent in a room or to a
# specific user.
# - one for ratelimiting 3PID invites (i.e. invites sent to a third-party ID
# such as an email address or a phone number) based on the account that's
# sending the invite.
#
# The defaults are as shown below.
#
@@ -217,6 +228,10 @@ class RatelimitConfig(Config):
# per_user:
# per_second: 0.003
# burst_count: 5
#
#rc_third_party_invite:
# per_second: 0.2
# burst_count: 10
# Ratelimiting settings for incoming federation
#

12
synapse/config/registration.py
View File

@@ -190,6 +190,8 @@ class RegistrationConfig(Config):
# The success template used during fallback auth.
self.fallback_success_template = self.read_template("auth_success.html")
self.inhibit_user_in_use_error = config.get("inhibit_user_in_use_error", False)
def generate_config_section(self, generate_secrets=False, **kwargs):
if generate_secrets:
registration_shared_secret = 'registration_shared_secret: "%s"' % (
@@ -446,6 +448,16 @@ class RegistrationConfig(Config):
# Defaults to true.
#
#auto_join_rooms_for_guests: false
# Whether to inhibit errors raised when registering a new account if the user ID
# already exists. If turned on, that requests to /register/available will always
# show a user ID as available, and Synapse won't raise an error when starting
# a registration with a user ID that already exists. However, Synapse will still
# raise an error if the registration completes and the username conflicts.
#
# Defaults to false.
#
#inhibit_user_in_use_error: true
"""
% locals()
)

96
synapse/config/server.py
View File

@@ -179,7 +179,6 @@ KNOWN_RESOURCES = {
"openid",
"replication",
"static",
"webclient",
}
@@ -200,8 +199,8 @@ class HttpListenerConfig:
"""Object describing the http-specific parts of the config of a listener"""
x_forwarded: bool = False
resources: List[HttpResourceConfig] = attr.ib(factory=list)
additional_resources: Dict[str, dict] = attr.ib(factory=dict)
resources: List[HttpResourceConfig] = attr.Factory(list)
additional_resources: Dict[str, dict] = attr.Factory(dict)
tag: Optional[str] = None
@@ -259,7 +258,6 @@ class ServerConfig(Config):
raise ConfigError(str(e))
self.pid_file = self.abspath(config.get("pid_file"))
self.web_client_location = config.get("web_client_location", None)
self.soft_file_limit = config.get("soft_file_limit", 0)
self.daemonize = config.get("daemonize")
self.print_pidfile = config.get("print_pidfile")
@@ -490,6 +488,19 @@ class ServerConfig(Config):
# events with profile information that differ from the target's global profile.
self.allow_per_room_profiles = config.get("allow_per_room_profiles", True)
# The maximum size an avatar can have, in bytes.
self.max_avatar_size = config.get("max_avatar_size")
if self.max_avatar_size is not None:
self.max_avatar_size = self.parse_size(self.max_avatar_size)
# The MIME types allowed for an avatar.
self.allowed_avatar_mimetypes = config.get("allowed_avatar_mimetypes")
if self.allowed_avatar_mimetypes and not isinstance(
self.allowed_avatar_mimetypes,
list,
):
raise ConfigError("allowed_avatar_mimetypes must be a list")
self.listeners = [parse_listener_def(x) for x in config.get("listeners", [])]
# no_tls is not really supported any more, but let's grandfather it in
@@ -506,8 +517,13 @@ class ServerConfig(Config):
l2.append(listener)
self.listeners = l2
if not self.web_client_location:
_warn_if_webclient_configured(self.listeners)
self.web_client_location = config.get("web_client_location", None)
# Non-HTTP(S) web client location is not supported.
if self.web_client_location and not (
self.web_client_location.startswith("http://")
or self.web_client_location.startswith("https://")
):
raise ConfigError("web_client_location must point to a HTTP(S) URL.")
self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
self.gc_seconds = self.read_gc_intervals(config.get("gc_min_interval", None))
@@ -635,19 +651,6 @@ class ServerConfig(Config):
False,
)
# List of users trialing the new experimental default push rules. This setting is
# not included in the sample configuration file on purpose as it's a temporary
# hack, so that some users can trial the new defaults without impacting every
# user on the homeserver.
users_new_default_push_rules: list = (
config.get("users_new_default_push_rules") or []
)
if not isinstance(users_new_default_push_rules, list):
raise ConfigError("'users_new_default_push_rules' must be a list")
# Turn the list into a set to improve lookup speed.
self.users_new_default_push_rules: set = set(users_new_default_push_rules)
# Whitelist of domain names that given next_link parameters must have
next_link_domain_whitelist: Optional[List[str]] = config.get(
"next_link_domain_whitelist"
@@ -793,13 +796,7 @@ class ServerConfig(Config):
#
pid_file: %(pid_file)s
# The absolute URL to the web client which /_matrix/client will redirect
# to if 'webclient' is configured under the 'listeners' configuration.
#
# This option can be also set to the filesystem path to the web client
# which will be served at /_matrix/client/ if 'webclient' is configured
# under the 'listeners' configuration, however this is a security risk:
# https://github.com/matrix-org/synapse#security-note
# The absolute URL to the web client which / will redirect to.
#
#web_client_location: https://riot.example.com/
@@ -883,7 +880,7 @@ class ServerConfig(Config):
# The default room version for newly created rooms.
#
# Known room versions are listed here:
# https://matrix.org/docs/spec/#complete-list-of-room-versions
# https://spec.matrix.org/latest/rooms/#complete-list-of-room-versions
#
# For example, for room version 1, default_room_version should be set
# to "1".
@@ -1011,8 +1008,6 @@ class ServerConfig(Config):
# static: static resources under synapse/static (/_matrix/static). (Mostly
# useful for 'fallback authentication'.)
#
# webclient: A web client. Requires web_client_location to be set.
#
listeners:
# TLS-enabled listener: for when matrix traffic is sent directly to synapse.
#
@@ -1168,6 +1163,20 @@ class ServerConfig(Config):
#
#allow_per_room_profiles: false
# The largest allowed file size for a user avatar. Defaults to no restriction.
#
# Note that user avatar changes will not work if this is set without
# using Synapse's media repository.
#
#max_avatar_size: 10M
# The MIME types allowed for user avatars. Defaults to no restriction.
#
# Note that user avatar changes will not work if this is set without
# using Synapse's media repository.
#
#allowed_avatar_mimetypes: ["image/png", "image/jpeg", "image/gif"]
# How long to keep redacted events in unredacted form in the database. After
# this period redacted events get replaced with their redacted form in the DB.
#
@@ -1337,11 +1346,16 @@ def parse_listener_def(listener: Any) -> ListenerConfig:
http_config = None
if listener_type == "http":
try:
resources = [
HttpResourceConfig(**res) for res in listener.get("resources", [])
]
except ValueError as e:
raise ConfigError("Unknown listener resource") from e
http_config = HttpListenerConfig(
x_forwarded=listener.get("x_forwarded", False),
resources=[
HttpResourceConfig(**res) for res in listener.get("resources", [])
],
resources=resources,
additional_resources=listener.get("additional_resources", {}),
tag=listener.get("tag"),
)
@@ -1349,24 +1363,6 @@ def parse_listener_def(listener: Any) -> ListenerConfig:
return ListenerConfig(port, bind_addresses, listener_type, tls, http_config)
NO_MORE_WEB_CLIENT_WARNING = """
Synapse no longer includes a web client. To enable a web client, configure
web_client_location. To remove this warning, remove 'webclient' from the 'listeners'
configuration.
"""
def _warn_if_webclient_configured(listeners: Iterable[ListenerConfig]) -> None:
for listener in listeners:
if not listener.http_options:
continue
for res in listener.http_options.resources:
for name in res.names:
if name == "webclient":
logger.warning(NO_MORE_WEB_CLIENT_WARNING)
return
_MANHOLE_SETTINGS_SCHEMA = {
"type": "object",
"properties": {

24
synapse/config/workers.py
View File

@@ -51,12 +51,12 @@ def _instance_to_list_converter(obj: Union[str, List[str]]) -> List[str]:
return obj
@attr.s
@attr.s(auto_attribs=True)
class InstanceLocationConfig:
"""The host and port to talk to an instance via HTTP replication."""
host = attr.ib(type=str)
port = attr.ib(type=int)
host: str
port: int
@attr.s
@@ -77,34 +77,28 @@ class WriterLocations:
can only be a single instance.
"""
events = attr.ib(
events: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)
typing = attr.ib(
typing: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)
to_device = attr.ib(
to_device: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)
account_data = attr.ib(
account_data: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)
receipts = attr.ib(
receipts: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)
presence = attr.ib(
presence: List[str] = attr.ib(
default=["master"],
type=List[str],
converter=_instance_to_list_converter,
)

Some files were not shown because too many files have changed in this diff Show More