| 20 Sep 2024 |
 andi- | works?! | 10:26:05 |
 Andreas Rammhold | seems like it | 10:28:22 |
 Yorusaka Miyabi | In reply to @andir:nixos.dev
seems like it Indeed. | 10:29:06 |
| andi- | Piping the documentation through a regex isn't ideal but well... 2nd try produced the right rules... | 10:29:17 |
| 24 Sep 2024 |
| Andreas Rammhold | Any more issues that I might have missed? | 09:04:37 |
| Yorusaka Miyabi | In reply to @andir:nixos.dev
Any more issues that I might have missed? Currently none, BTW if you only delete media files, Synapse would still believe the media is in the remote media cache, only failed to found in the cache and report error to clients. | 09:06:39 |
| andi- | Yeah, I run a purge script less frequent than the actual cleanup. Can probably tweak the timining of that. I was a stop gap because at times people consumed tons of gigabytes of images every day... | 09:07:26 |
| 29 Sep 2024 |
| Yorusaka Miyabi | andi- BTW, I've seen reports that https://logs.nixos.dev is mostly broken
| 06:42:31 |
| andi- | That shitty go program strikes again. Iām not home until the evening. I used to restart it once an hour with a hackt loop runnning in tmux š¤£ have to make more permanent attempts | 06:48:22 |
| andi- | It kicks the bucket after enough requests. | 06:48:31 |
| 4 Oct 2024 |
|  dusk changed their profile picture. | 04:41:34 |
| 17 Oct 2024 |
| Yorusaka Miyabi | Appeareantly I lost many events in !cacbMwUwsLZ6GKac:nichi.co š¤ | 13:55:07 |
| Yorusaka Miyabi | In reply to @shadowrz:nixos.dev
Appeareantly I lost many events in !cacbMwUwsLZ6GKac:nichi.co š¤ For example Event ID $RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34 is strangely unavaliable from nixos.dev Cc @andi:kack.it | 14:20:23 |
| andi- | no idea why that would be or how to fix that now.. can't see anything on the server | 14:20:59 |
| Yorusaka Miyabi | In reply to @andi:kack.it
no idea why that would be or how to fix that now.. can't see anything on the server ? Synapse emited no warnings ? | 14:37:47 |
| Yorusaka Miyabi | In reply to @andi:kack.it
no idea why that would be or how to fix that now.. can't see anything on the server * | 14:37:52 |
| Yorusaka Miyabi | In reply to @shadowrz:nixos.dev
? Synapse emited no warnings or errors? Also I did another one, https://matrix.nixos.dev/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac%3Anichi.co/event/%24RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34 if you want to troubleshoot | 14:39:16 |
| Yorusaka Miyabi | Syncing that room is also missing many events and only few goes in, unsure why it would happen | 14:45:48 |
| Andreas Rammhold |
Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
| 14:45:49 |
| Yorusaka Miyabi | Redacted or Malformed Event | 14:46:20 |
| Yorusaka Miyabi | In reply to @andir:nixos.dev
Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
* | 14:46:23 |
| Yorusaka Miyabi | In reply to @andir:nixos.dev
Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8Oct 17 14:34:43 matrix synapse-client_worker2[1163]: synapse.http.server: [PUT-5896459] <XForwardedForRequest at 0x7ff39a3da050 method='PUT' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 405 - Unrecognized request
No not this, I managed to made the HTTP method wrong | 14:46:31 |
| Andreas Rammhold |
Oct 17 14:37:00 matrix synapse-client_worker2[1163]: synapse.http.server: [GET-5896991] <XForwardedForRequest at 0x7ff3cf4a33d0 method='GET' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac%3Anichi.co/event/%24RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 404 - Event not found.
this?
| 14:46:52 |
| Yorusaka Miyabi | In reply to @andir:nixos.dev
Oct 17 14:37:00 matrix synapse-client_worker2[1163]: synapse.http.server: [GET-5896991] <XForwardedForRequest at 0x7ff3cf4a33d0 method='GET' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac%3Anichi.co/event/%24RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 404 - Event not found.
this?
Yes, like this | 14:47:26 |
| Yorusaka Miyabi | It also affected the public log on https://logs.nixos.dev/room/!cacbMwUwsLZ6GKac:nichi.co/ where lots events are missing from sync too | 14:49:58 |
| andi- | Oct 17 13:14:23 matrix synapse-federation_sender2[1165]: synapse.federation.federation_server: [_process_incoming_pdus_in_room_inner-131398] handling received PDU in room !cacbMwUwsLZ6GKac:nichi.co: <FrozenEventV3 event_id=$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34, type=m.room.message, state_key=None, outlier=False>
Oct 17 13:14:23 matrix synapse-federation_sender2[1165]: synapse.handlers.federation_event: [_process_incoming_pdus_in_room_inner-131398-$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34] While checking auth of <FrozenEventV3 event_id=$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34, type=m.room.message, state_key=None, outlier=False> against auth_events: 403: During auth for event $RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34: found rejected event $gptoDaHlOKSxtvJP713Mc0ywTnQDvcDy400d0jCAtfo in the state
Oct 17 13:14:23 matrix synapse-federation_sender2[1165]: synapse.http.client: [_process_incoming_pdus_in_room_inner-131398-$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34] Received response to POST synapse-replication://master/_synapse/replication/fed_send_events/uLADjbYQlB: 200
Oct 17 13:14:23 matrix synapse-federation_sender2[1165]: synapse.replication.tcp.client: [_process_incoming_pdus_in_room_inner-131398-$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34] Waiting for repl stream 'events' to reach 21684842 (master); currently at: 21684841
Oct 17 13:14:23 matrix synapse-federation_sender2[1165]: synapse.replication.tcp.client: [_process_incoming_pdus_in_room_inner-131398-$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34] Finished waiting for repl stream 'events' to reach 21684842 (master)
Oct 17 13:42:31 matrix synapse-client_worker2[1163]: synapse.http.server: [GET-5884919] <XForwardedForRequest at 0x7ff3d7824050 method='GET' uri='/_matrix/client/v3/rooms/!cacbMwUwsLZ6GKac:nichi.co/event/$RKy3adIpkxQtqI0TwSFLMMOg8-mYhYARrXWVfrajz34' clientproto='HTTP/1.1' site='7501'> SynapseError: 404 - Event not found.
| 14:55:37 |
| andi- | restarted synapse (and the workers) to purge any potential wrong in-memory caches that might exist | 14:57:33 |
| andi- | In reply to @shadowrz:nixos.dev
It also affected the public log on https://logs.nixos.dev/room/!cacbMwUwsLZ6GKac:nichi.co/ where lots events are missing from sync too What timeframe? For me that looked fine | 14:57:43 |
| Yorusaka Miyabi | In reply to @andi:kack.it
What timeframe? For me that looked fine At least for a week but even earlier, I have confirmed that there should be even more messages, it's actually a room bridged with Telegram, and vast majority messaged are bridged to Matrix from Telegram via a bridge on the nichi.co homeserver | 14:59:26 |
| Yorusaka Miyabi | In reply to @andi:kack.it
What timeframe? For me that looked fine * | 14:59:36 |
