If the script fails (or is CTRL-C'ed) between porting some of the events table and copying of the sequences then the port script will immediately die if run again due to the postgres DB having inconsistencies between sequences and tables.
The fix is to move the porting of sequences to before porting the tables, so that there is never a period where the Postgres DB is inconsistent. To do that we need to change how we port the sequences so that it calculates the values from the SQLite DB rather than the Postgres DB.
Fixes#8619
* commit 'f43c66d23':
Add support for running Complement against the local checkout (#8317)
Filter out appservices from mau count (#8404)
Only assert valid next_link params when provided (#8417)
Add metrics to track success/otherwise of replication requests (#8406)
Fix handling of connection timeouts in outgoing http requests (#8400)
Changelog
Don't check whether a 3pid is allowed to register during password reset
Add checks for postgres sequence consistency (#8402)
Create a mechanism for marking tests "logcontext clean" (#8399)
Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410)
A pair of tiny cleanups in the federation request code. (#8401)
typo
* commit '31acc5c30':
Escape the error description on the sso_error template. (#8405)
Fix occasional "Re-starting finished log context" from keyring (#8398)
Allow existing users to login via OpenID Connect. (#8345)
Fix schema delta for servers that have not backfilled (#8396)
Fix MultiWriteIdGenerator's handling of restarts. (#8374)
s/URLs/variables in changelog
s/accidentally/incorrectly in changelog
Update changelog wording
Add type annotations to SimpleHttpClient (#8372)
Add new sequences to port DB script (#8387)
Add EventStreamPosition type (#8388)
Mark the shadow_banned column as boolean in synapse_port_db. (#8386)
* commit '920dd1083':
1.20.1
Mark the shadow_banned column as boolean in synapse_port_db. (#8386)
Hotfix: disable autoescape by default when rendering Jinja2 templates (#8394)
* commit '837293c31':
Remove obsolete __future__ imports (#8337)
Use admin_patterns for all admin APIs. (#8331)
Fix a potential bug of UnboundLocalError (#8329)
Switch metaclass initialization to python 3-compatible syntax (#8326)
Catch-up after Federation Outage (split, 4): catch-up loop (#8272)
Use slots in attrs classes where possible (#8296)
Fix typos in comments.
Add the topic and avatar to the room details admin API (#8305)
Improve SAML error messages (#8248)
Add experimental support for sharding event persister. Again. (#8294)
Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)
Use TLSv1.2 for fake servers in tests (#8208)
Add /_synapse/client to the reverse proxy docs (#8227)
Clean up `Notifier.on_new_room_event` code path (#8288)
I noticed in https://github.com/matrix-org/synapse/issues/8575 that the `end_error` variable in `synapse_port_db` is set to an `Exception`, even though later we expect it to be a `str`.
This PR simply casts an exception raised to a string. I'm doing this instead of having `end_error` be of type exception as we explicitly set `end_error` to a str here:
d25eb8f370/scripts/synapse_port_db (L542-L547)
This whole file could probably use some heavy refactoring, but until then at least this fix will prevent exception contents from being hidden from us and users.
* commit '3950ae51e':
Ensure that remove_pusher is always async (#7981)
Ensure the msg property of HttpResponseException is a string. (#7979)
Remove from the event_relations table when purging historical events. (#7978)
Add additional logging for SAML sessions. (#7971)
Add MSC reference to changelog for #7736
Re-implement unread counts (#7736)
Various improvements to the docs (#7899)
Convert storage layer to async/await. (#7963)
Add an option to disable purge in delete room admin API (#7964)
Move some log lines from default logger to sql/transaction loggers (#7952)
Use the JSON module from the std library instead of simplejson. (#7936)
Fix exit code for `check_line_terminators.sh` (#7970)
Option to allow server admins to join complex rooms (#7902)
Fix typo in metrics docs (#7966)
Add script for finding files with unix line terminators (#7965)
Convert the remaining media repo code to async / await. (#7947)
Convert a synapse.events to async/await. (#7949)
Convert groups and visibility code to async / await. (#7951)
Convert push to async/await. (#7948)
This is so we can tell what is going on when things are taking a while to start up.
The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
This table was created in #8034 (1.20.0). It references
`ui_auth_sessions`, which is ignored, so this one should be too.
Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
* commit 'de119063f': (31 commits)
Convert room list handler to async/await. (#7912)
Element CSS and logo in email templates (#7919)
Lint the contrib/ directory in CI and linting scripts, add synctl to linting script (#7914)
Remove unused code from synapse.logging.utils. (#7897)
Fix a typo in the sample config. (#7890)
Fix deprecation warning: import ABC from collections.abc (#7892)
Change sample config's postgres user to synapse_user (#7889)
Fix deprecation warning due to invalid escape sequences (#7895)
Remove Ubuntu Eoan that is now EOL (#7888)
Fix the trace function for async functions. (#7872)
Add help for creating a user via docker (#7885)
Switch to Debian:Slim from Alpine for the docker image (#7839)
Stop using 'device_max_stream_id' (#7882)
Fix TypeError in synapse.notifier (#7880)
Add a default limit (of 100) to get/sync operations. (#7858)
Change "unknown room ver" logging to warning. (#7881)
Convert device handler to async/await (#7871)
Convert synapse.app to async/await. (#7868)
Convert _base, profile, and _receipts handlers to async/await (#7860)
Add admin endpoint to get members in a room. (#7842)
...
* commit 'dc80a0762':
1.16.0rc1
Back out MSC2625 implementation (#7761)
Additional configuration options for auto-join rooms (#7763)
Add some metrics for inbound and outbound federation processing times (#7755)
Explain the purpose of the "tests" conditional dependency requirement (#7751)
Add another yield point to state res v2 (#7746)
Move flake8 to end. Don't exit script on failure (#7738)
Make tox actions work on Debian 10 (#7703)
Yield during large v2 state res. (#7735)
add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675)
Set Content-Length for Metrics requests (#7730)
Sync ignored table names in synapse_port_db to current database schema (#7717)
Allow local media to be marked as safe from being quarantined. (#7718)
Convert directory handler to async/await (#7727)
Speed up state res v2 across large state differences. (#7725)
* 'release-v1.14.0' of github.com:matrix-org/synapse: (108 commits)
Fix typo in PR link
Update debian changelog
1.14.0
Improve changelog wording
1.14.0rc2
Fix sample config docs error (#7581)
Fix up comments
Fix specifying cache factors via env vars with * in name. (#7580)
Don't apply cache factor to event cache. (#7578)
Ensure ReplicationStreamer is always started when replication enabled. (#7579)
Remove the changes to the debian changelog
Not full release yet, this is rc1
Merge event persistence move changelog entries
More changelog fix
Changelog fixes
1.14.0
Replace device_27_unique_idx bg update with a fg one (#7562)
Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563)
Fix recording of federation stream token (#7564)
Simplify reap_monthly_active_users (#7558)
...
The old statement returned `None` for such a `password_config` (like the one
created on first run), thus retrieval of the `pepper` key failed with
`AttributeError`.
Fixes#5315
Signed-off-by: Ivan Vilata i Balaguer <ivan@selidor.net>
* Raise an exception if there are pending background updates
So we return with a non-0 code
* Changelog
* Port synapse_port_db to async/await
* Port update_database to async/await
* Add version string to mocked homeservers
* Remove unused imports
* Convert overseen bits to async/await
* Fixup logging contexts
* Fix imports
* Add a way to print an error without raising an exception
* Incorporate review
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
