kaslo/synapse
1
0
Fork 0
Code Wiki Activity
20,904 Commits 1,047 Branches 805 Tags
262007f049e5c17d795890ffe00218b9c2fe1acf
Commit Graph

132 Commits

Author SHA1 Message Date
Brendan Abolivier
262007f049 Merge tag 'v1.39.0' into babolivier/dinsic_1.41.0 
Synapse 1.39.0 (2021-07-29)
===========================

No significant changes.

Synapse 1.39.0rc3 (2021-07-28)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477))
- Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485))
- Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486))

Internal Changes
----------------

- Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461))
- Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))

Synapse 1.39.0rc2 (2021-07-22)
==============================

This release also includes the changes in v1.38.1.

Internal Changes
----------------

- Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))

Synapse 1.39.0rc1 (2021-07-20)
==============================

The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information.

Features
--------

- Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884))
- The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305))
- Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427))
- Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360))

Bugfixes
--------

- Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433))
- Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250))
- Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276))
- Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317))
- Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343))
- Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344))
- Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355))
- Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359))
- Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367))
- Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414))
- Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404))

Improved Documentation
----------------------

- Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971))
- Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313))
- Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337))
- Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368))
- Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399))
- Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395))
- Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396))

Deprecations and Removals
-------------------------

- Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721))
- The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386))

Internal Changes
----------------

- Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289))
- Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315))
- Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316))
- Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324))
- Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357))
- Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348))
- Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349))
- Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418))
- Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383))
- Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385))
- When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391))
- Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393))
- Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398))
- Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421))
- Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
 2021-09-01 11:26:23 +01:00
Brendan Abolivier
1d4f5c34d8 Merge tag 'v1.33.0' into babolivier/dinsic_1.41.0 
Synapse 1.33.0 (2021-05-05)
===========================

Features
--------

- Build Debian packages for Ubuntu 21.04 (Hirsute Hippo). ([\#9909](https://github.com/matrix-org/synapse/issues/9909))

Synapse 1.33.0rc2 (2021-04-29)
==============================

Bugfixes
--------

- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))

Synapse 1.33.0rc1 (2021-04-28)
==============================

Features
--------

- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))

Bugfixes
--------

- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))

Improved Documentation
----------------------

- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))

Internal Changes
----------------

- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
 2021-08-31 14:53:31 +01:00
Brendan Abolivier
9de3991b9e Merge tag 'v1.32.0' into babolivier/dinsic_1.41.0 
Synapse 1.32.0 (2021-04-20)
===========================

**Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+.

This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities.

This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date.

Bugfixes
--------

- Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](https://github.com/matrix-org/synapse/issues/9829))

Synapse 1.32.0rc1 (2021-04-13)
==============================

Features
--------

- Add a Synapse module for routing presence updates between users. ([\#9491](https://github.com/matrix-org/synapse/issues/9491))
- Add an admin API to manage ratelimit for a specific user. ([\#9648](https://github.com/matrix-org/synapse/issues/9648))
- Include request information in structured logging output. ([\#9654](https://github.com/matrix-org/synapse/issues/9654))
- Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](https://github.com/matrix-org/synapse/issues/9691))
- Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](https://github.com/matrix-org/synapse/issues/9700))
- Add experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9717](https://github.com/matrix-org/synapse/issues/9717), [\#9735](https://github.com/matrix-org/synapse/issues/9735))
- Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](https://github.com/matrix-org/synapse/issues/9710))
- Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](https://github.com/matrix-org/synapse/issues/9766))

Bugfixes
--------

- Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](https://github.com/matrix-org/synapse/issues/8926))
- Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](https://github.com/matrix-org/synapse/issues/9711))
- Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](https://github.com/matrix-org/synapse/issues/9725))
- Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](https://github.com/matrix-org/synapse/issues/9770))
- Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](https://github.com/matrix-org/synapse/issues/9780))

Updates to the Docker image
---------------------------

- Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](https://github.com/matrix-org/synapse/issues/9765))

Improved Documentation
----------------------

- Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](https://github.com/matrix-org/synapse/issues/9719))

Deprecations and Removals
-------------------------

- Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](https://github.com/matrix-org/synapse/issues/9401))
- Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](https://github.com/matrix-org/synapse/issues/9548))

Internal Changes
----------------

- Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](https://github.com/matrix-org/synapse/issues/9718))
- Experiment with GitHub Actions for CI. ([\#9661](https://github.com/matrix-org/synapse/issues/9661))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](https://github.com/matrix-org/synapse/issues/9682))
- Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](https://github.com/matrix-org/synapse/issues/9685))
- Improve Jaeger tracing for `to_device` messages. ([\#9686](https://github.com/matrix-org/synapse/issues/9686))
- Add release helper script for automating part of the Synapse release process. ([\#9713](https://github.com/matrix-org/synapse/issues/9713))
- Add type hints to expiring cache. ([\#9730](https://github.com/matrix-org/synapse/issues/9730))
- Convert various testcases to `HomeserverTestCase`. ([\#9736](https://github.com/matrix-org/synapse/issues/9736))
- Start linting mypy with `no_implicit_optional`. ([\#9742](https://github.com/matrix-org/synapse/issues/9742))
- Add missing type hints to federation handler and server. ([\#9743](https://github.com/matrix-org/synapse/issues/9743))
- Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](https://github.com/matrix-org/synapse/issues/9753))
- Fix incompatibility with `tox` 2.5. ([\#9769](https://github.com/matrix-org/synapse/issues/9769))
- Enable Complement tests for [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946): Spaces Summary API. ([\#9771](https://github.com/matrix-org/synapse/issues/9771))
- Use mock from the standard library instead of a separate package. ([\#9772](https://github.com/matrix-org/synapse/issues/9772))
- Update Black configuration to target Python 3.6. ([\#9781](https://github.com/matrix-org/synapse/issues/9781))
- Add option to skip unit tests when building Debian packages. ([\#9793](https://github.com/matrix-org/synapse/issues/9793))
 2021-08-31 11:40:27 +01:00
Dirk Klimpel
c7603af1d0 Allow providing credentials to http_proxy (#10360) 2021-07-15 10:37:08 +01:00
Jonathan de Jong
89cfc3dd98 [pyupgrade] tests/ (#10347) 2021-07-13 11:43:15 +01:00
Richard van der Hoff
3ff2251754 Improved validation for received requests (#9817) 
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
 2021-04-23 19:20:44 +01:00
Andrew Morgan
8cb8c604e4 Merge commit '55da8df07' into anoa/dinsic_release_1_31_0 2021-04-23 17:28:38 +01:00
Andrew Morgan
5d4591515e Merge commit 'f00c4e7af' into anoa/dinsic_release_1_31_0 2021-04-23 16:18:51 +01:00
Andrew Morgan
7ce8d7877e Merge commit 'e12077a78' into anoa/dinsic_release_1_31_0 2021-04-23 15:44:57 +01:00
Andrew Morgan
2036503c9e Merge commit 'd9f1dccba' into anoa/dinsic_release_1_31_0 2021-04-23 13:39:50 +01:00
Andrew Morgan
458add9e29 Merge commit '8ec221710' into anoa/dinsic_release_1_31_0 2021-04-23 12:22:35 +01:00
Andrew Morgan
476cc03d7a Merge commit 'd2f0ec12d' into anoa/dinsic_release_1_31_0 2021-04-23 12:22:28 +01:00
Richard van der Hoff
51a20914a8 Limit the size of HTTP responses read over federation. (#9833) 2021-04-23 11:08:41 +01:00
Andrew Morgan
15f31e1ca9 Merge commit '0cd2938bc' into anoa/dinsic_release_1_31_0 2021-04-22 18:26:10 +01:00
Andrew Morgan
a31eb4a9fd Merge commit '3e4cdfe5d' into anoa/dinsic_release_1_31_0 2021-04-22 16:34:35 +01:00
Andrew Morgan
77f011b8fd Merge commit '98a64b7f7' into anoa/dinsic_release_1_31_0 2021-04-22 16:21:11 +01:00
Andrew Morgan
bae39a4777 Merge commit '3dd6ba135' into anoa/dinsic_release_1_31_0 2021-04-21 16:17:14 +01:00
Andrew Morgan
83c1c8a56b Merge commit 'ff5c4da12' into anoa/dinsic_release_1_31_0 2021-04-20 16:41:22 +01:00
Andrew Morgan
ec2cab331d Merge commit 'bd30cfe86' into anoa/dinsic_release_1_31_0 2021-04-16 15:45:40 +01:00
Jonathan de Jong
4b965c862d Remove redundant "coding: utf-8" lines (#9786) 
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
 2021-04-14 15:34:27 +01:00
Patrick Cloke
0b3112123d Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Jonathan de Jong
2ca4e349e9 Bugbear: Add Mutable Parameter fixes (#9682) 
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-04-08 22:38:54 +01:00
Andrew Morgan
0060eb332c Port "Allow providing credentials to HTTPS_PROXY (#9657)" from mainline (#95) 
* Allow providing credentials to HTTPS_PROXY (#9657)

Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.

* lint
 2021-03-23 15:35:38 +01:00
Andrew Morgan
232b324c38 Port "Add support for no_proxy and case insensitive env variables" from mainline to dinsic (#93) 
This PR is simply porting https://github.com/matrix-org/synapse/pull/9372 to dinsic.

I also had to bring in https://github.com/matrix-org/synapse/pull/8821 and https://github.com/matrix-org/synapse/pull/9084 for this code to work properly - a sign that we should merge mainline into dinsic again soon.
 2021-03-22 17:48:42 +00:00
Andrew Morgan
5b268997bd Allow providing credentials to HTTPS_PROXY (#9657) 
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
 2021-03-22 17:20:47 +00:00
Patrick Cloke
e55bd0e110 Add tests for blacklisting reactor/agent. (#9563) 2021-03-11 09:15:22 -05:00
Patrick Cloke
16ec8c3272 (Hopefully) stop leaking file descriptors in media repo. (#9497) 
By consuming the response if the headers imply that the
content is too large.
 2021-03-01 12:45:00 -05:00
Tim Leung
ddb240293a Add support for no_proxy and case insensitive env variables (#9372) 
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
 2021-02-26 17:37:57 +00:00
Patrick Cloke
8ec2217103 Reduce the memory usage of previewing media files. (#9421) 
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
 2021-02-18 09:01:29 -05:00
Eric Eastwood
0a00b7ff14 Update black, and run auto formatting over the codebase (#9381) 
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
 2021-02-16 22:32:34 +00:00
Richard van der Hoff
0cd2938bc8 Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke
74dd906041 Avoid raising the body exceeded error multiple times. (#9108) 
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
 2021-01-15 11:00:13 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Marcus
e385c8b473 Don't apply the IP range blacklist to proxy connections (#9084) 
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
 2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Andrew Morgan
5c51806d1d Merge commit 'acfe3b306' into anoa/dinsic_release_1_23_1 2020-12-31 15:22:27 +00:00
Andrew Morgan
6ee9980843 Merge commit '129ae841e' into anoa/dinsic_release_1_23_1 2020-12-31 14:55:29 +00:00
Patrick Cloke
ff5c4da128 Add a maximum size for well-known lookups. (#8950) 2020-12-16 17:25:24 -05:00
Richard van der Hoff
394516ad1b Remove spurious "SynapseRequest" result from `make_request" 
This was never used, so let's get rid of it.
 2020-12-15 22:35:40 +00:00
Richard van der Hoff
7eebe4b3fc Replace request.code with channel.code 
The two are equivalent, but really we want to check the HTTP result that got
returned to the channel, not the code that the Request object *intended* to
return to the channel.
 2020-12-15 22:32:12 +00:00
Patrick Cloke
30fba62108 Apply an IP range blacklist to push and key revocation requests. (#8821) 
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
 2020-12-02 11:09:24 -05:00
Richard van der Hoff
be8fa65d0b Remove redundant calls to render() 2020-11-16 18:24:08 +00:00
Richard van der Hoff
cfd895a22e use global make_request() directly where we have a custom Resource 
Where we want to render a request against a specific Resource, call the global
make_request() function rather than the one in HomeserverTestCase, allowing us
to pass in an appropriate `Site`.
 2020-11-15 23:09:03 +00:00
Andrew Morgan
ab26b5894a Merge commit 'f43c66d23' into anoa/dinsic_release_1_21_x 
* commit 'f43c66d23':
  Add support for running Complement against the local checkout (#8317)
  Filter out appservices from mau count (#8404)
  Only assert valid next_link params when provided (#8417)
  Add metrics to track success/otherwise of replication requests (#8406)
  Fix handling of connection timeouts in outgoing http requests (#8400)
  Changelog
  Don't check whether a 3pid is allowed to register during password reset
  Add checks for postgres sequence consistency (#8402)
  Create a mechanism for marking tests "logcontext clean" (#8399)
  Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410)
  A pair of tiny cleanups in the federation request code. (#8401)
  typo
 2020-10-21 14:53:14 +01:00
Andrew Morgan
7176832124 Merge commit '837293c31' into anoa/dinsic_release_1_21_x 
* commit '837293c31':
  Remove obsolete __future__ imports (#8337)
  Use admin_patterns for all admin APIs. (#8331)
  Fix a potential bug of UnboundLocalError (#8329)
  Switch metaclass initialization to python 3-compatible syntax (#8326)
  Catch-up after Federation Outage (split, 4): catch-up loop (#8272)
  Use slots in attrs classes where possible (#8296)
  Fix typos in comments.
  Add the topic and avatar to the room details admin API (#8305)
  Improve SAML error messages (#8248)
  Add experimental support for sharding event persister. Again. (#8294)
  Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)
  Use TLSv1.2 for fake servers in tests (#8208)
  Add /_synapse/client to the reverse proxy docs (#8227)
  Clean up `Notifier.on_new_room_event` code path (#8288)
 2020-10-20 19:51:28 +01:00
Andrew Morgan
e13218756c Merge commit '2832ef5bb' into anoa/dinsic_release_1_21_x 
* commit '2832ef5bb':
  1.20.0rc3
  Fix the exception that is raised when invalid JSON is encountered. (#8291)
  Remove shared rooms info from upgrade/workers doc as it's still experimental (#8290)
 2020-10-20 19:28:12 +01:00
Andrew Morgan
47d53b32e3 Merge commit '17fa4c7ca' into anoa/dinsic_release_1_21_x 
* commit '17fa4c7ca':
  Catch up after Federation Outage (split, 2): Track last successful stream ordering after transmission (#8247)
  Catch-up after Federation Outage (split, 1) (#8230)
  Fix type signature in simple_select_one_onecol and friends (#8241)
  Stop sub-classing object (#8249)
 2020-10-20 18:09:59 +01:00
Andrew Morgan
059e0fdedf Merge commit '5bf8e5f55' into anoa/dinsic_release_1_21_x 
* commit '5bf8e5f55':
  Convert the well known resolver to async (#8214)
  Convert additional databases to async/await part 2 (#8200)
  Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203)
  Do not install setuptools 50.0. (#8212)
  Move and rename `get_devices_with_keys_by_user` (#8204)
  Rename `get_e2e_device_keys` to better reflect its purpose (#8205)
  Add a comment about _LimitedHostnameResolver
 2020-10-20 17:42:29 +01:00
Andrew Morgan
df77da8b4d Merge commit 'a466b6797' into anoa/dinsic_release_1_21_x 
* commit 'a466b6797':
  Reduce run-times of tests by advancing the reactor less (#7757)
  Update debian systemd service to use Type=notify (#8169)
  Remove remaining is_guest argument uses from get_room_data calls (#8181)
  Do not propagate typing notifications from shadow-banned users. (#8176)
  Remove unused parameter from, and add safeguard in, get_room_data (#8174)
  Add required Debian dependencies to allow docker builds on the arm platform (#8144)
  Allow running mypy directly. (#8175)
  Update the test federation client to handle streaming responses (#8130)
  Do not propagate profile changes of shadow-banned users into rooms. (#8157)
  Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator (#8171)
  Convert simple_select_one and simple_select_one_onecol to async (#8162)
 2020-10-20 17:13:42 +01:00
Andrew Morgan
4ae35a3c7b Merge commit 'c978f6c45' into anoa/dinsic_release_1_21_x 
* commit 'c978f6c45':
  Convert federation client to async/await. (#7975)
 2020-10-16 17:08:04 +01:00