| 7 Oct 2025 |
 raitobezarius (DECT: 7248) | gerrit.lix.systems is back up again | 17:56:24 |
| raitobezarius (DECT: 7248) | If you are curious as what happened (my soul is dried for now), you can enjoy this story: https://zulip.afnix.fr/#narrow/channel/4-infra/topic/gerrit.2Elix.2Esystems.20migration/near/7026 | 17:57:16 |
| raitobezarius (DECT: 7248) | TL;DR: H2 database which contains "mark files as reviewed" self corrupted on each restarts, we observed CL which were merged to be "unmerged", leading us to worry about this data corruption. We realized later on that rsync didn't synchronize the actual state files and the NoteDB which contained the source of truth on reviews. We used another approach and nuked the H2 database for now. | 17:58:07 |
| raitobezarius (DECT: 7248) | flokli: You will certainly be affected by this as you are on Gerrit 3.12 as well, H2 self corrupts after a certain scale: https://github.com/h2database/h2database/issues/4123 FYI | 17:59:41 |
| raitobezarius (DECT: 7248) | TL;DR for Lix developers: all "mark files as reviewed" information is gone for now. I deem this information to be minor, if you disagree, please let me know and I will forensics the old copy we have to restore the database. | 18:00:30 |
| raitobezarius (DECT: 7248) | All the rest of data is safe. | 18:00:36 |
 just1602 | I read the Zulip thread and things seems fixed, but on my side I get PR_CONNECT_RESET_ERROR when I try to access gerrit.lix.systems | 19:50:12 |
| just1602 | Oh it works on mobile, but not at work. It might be on my side | 19:52:03 |
| raitobezarius (DECT: 7248) | A redeployment happened at that moment | 19:52:55 |
| raitobezarius (DECT: 7248) | Do you still have the error at work? | 19:53:03 |
 Rutile (Commentator2.0) feel free to ping | i get an ominous "Server error" when trying to log in at gerrit | 19:53:17 |
| raitobezarius (DECT: 7248) | I know what's going on, give me a sec | 19:53:27 |
| raitobezarius (DECT: 7248) | Rutile (Commentator2.0) feel free to ping: can you give it a try again? | 19:55:30 |
| raitobezarius (DECT: 7248) | aaaaaaaa | 19:55:46 |
| Rutile (Commentator2.0) feel free to ping | same error | 19:55:48 |
| raitobezarius (DECT: 7248) | wait | 19:55:49 |
| raitobezarius (DECT: 7248) | it won't work | 19:55:50 |
| raitobezarius (DECT: 7248) | Rutile (Commentator2.0) feel free to ping: give it a try now | 19:57:13 |
| raitobezarius (DECT: 7248) | Please give it a try as well | 19:57:18 |
 Katalin 🔪 | still happens for me | 19:57:50 |
| Rutile (Commentator2.0) feel free to ping | negative | 19:57:51 |
| raitobezarius (DECT: 7248) | clearing some cache here and there | 19:58:07 |
| just1602 | Doesn't work neither | 19:58:16 |
| raitobezarius (DECT: 7248) | Give me 2 minutes or so | 19:59:26 |
| raitobezarius (DECT: 7248) | Alright, just1602 Rutile (Commentator2.0) feel free to ping Katalin 🔪 | 20:03:08 |
| raitobezarius (DECT: 7248) | Can you try again? | 20:03:10 |
| Rutile (Commentator2.0) feel free to ping | worked :D | 20:03:19 |
| Rutile (Commentator2.0) feel free to ping | thankuuuuu :3 | 20:03:23 |
| Katalin 🔪 | still broken for me | 20:03:43 |
| Rutile (Commentator2.0) feel free to ping | also, good job on cleaning this whole thing up and figuring out why it broke in the first place | 20:03:43 |
