| 6 Jun 2024 |
 K900 | Also btw I have way too much experience with cursed Python | 19:11:10 |
| K900 | So if you need cursed Python help, hit me up | 19:11:15 |
 jade_ | anyone wanna review a small pile of release engineering? :D https://gerrit.lix.systems/c/lix/+/1410 | 19:46:51 |
 Qyriad | Once we figure out why buildbot is down >.< | 19:54:04 |
 raitobezarius |
Jun 06 19:29:42 buildbot systemd[1]: postgresql.service: systemd-oomd killed 11 process(es) in this unit.
| 20:02:09 |
| jade_ | oh boy | 20:02:16 |
| jade_ | big ouchie | 20:02:24 |
| raitobezarius | we have 32GB of RAM on the machine | 20:02:34 |
| raitobezarius | I can bump it to 64GB of RAM | 20:02:37 |
| jade_ | we should really make buildbot more killable and shove some guaranteed ram at postgres | 20:02:39 |
| jade_ | that would be the actually reasonable move probably | 20:02:50 |
| raitobezarius | okie dokie | 20:02:57 |
| jade_ | needs more MemoryMin | 20:02:59 |
| Qyriad | It's back up now at least; @jade_ would you mind amending the first commit in that series and then rebasing or something? | 20:04:43 |
| Qyriad | Also I don't know if it's important but we're getting a bunch of these in the postgres logs now
postgresql.service[2288352]: [2288352] WARNING: database "postgres" has a collation version mismatch
postgresql.service[2288352]: [2288352] DETAIL: The database was created using collation version 2.38, but the operating system provides version 2.39.
postgresql.service[2288352]: [2288352] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
postgresql.service[2288353]: [2288353] WARNING: database "buildbot" has a collation version mismatch
postgresql.service[2288353]: [2288353] DETAIL: The database was created using collation version 2.38, but the operating system provides version 2.39.
postgresql.service[2288353]: [2288353] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE buildbot REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
| 20:05:46 |
| jade_ | oh ffs, i know how to fix her | 20:05:57 |
| jade_ | brb | 20:05:58 |
 kloenk | oh fun had my postgres not even starting sone time ago with that | 20:06:36 |
| jade_ | she should be fixed now | 20:06:39 |
| jade_ | same | 20:06:41 |
| Qyriad | Thank you | 20:07:12 |
 ⚠️ eldritch horrors operating in this area ⚠️ | oof yeah
we meant to fix that since we first saw it weeks(?) ago but kept forgetting >,< | 20:10:20 |
| K900 | In reply to @qyriad:katesiria.org
Also I don't know if it's important but we're getting a bunch of these in the postgres logs now
postgresql.service[2288352]: [2288352] WARNING: database "postgres" has a collation version mismatch
postgresql.service[2288352]: [2288352] DETAIL: The database was created using collation version 2.38, but the operating system provides version 2.39.
postgresql.service[2288352]: [2288352] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
postgresql.service[2288353]: [2288353] WARNING: database "buildbot" has a collation version mismatch
postgresql.service[2288353]: [2288353] DETAIL: The database was created using collation version 2.38, but the operating system provides version 2.39.
postgresql.service[2288353]: [2288353] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE buildbot REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
It's fine | 20:10:42 |
| K900 | Postgres collations are a little weird | 20:11:04 |
| K900 | But everything sane should be using C.UTF-8 anyway | 20:11:14 |
| Qyriad | Hm, the UI is up now but buildbot isn't actually building anything | 20:13:36 |
| Qyriad | Only thing of note we see in the journal is
nginx.service[2280711]: 2024/06/06 20:11:08 [error] 2280711#2280711: *545 [lua] openidc.lua:1484: authenticate(): request to the redirect_uri path but there's no session state found, client: 2a01:e0a:b99:6681:d68a:3bb6:a76e:50f4, server: buildbot.lix.systems, request: "GET /redirect_uri?state=b93666ecff611200b8c9bbb90530952e&session_state=86c9f164-921b-4ca4-95dd-68b3c4bb35c2&iss=https%3A%2F%2Fidentity.lix.systems%2Frealms%2Flix-project&code=9c2e30f5-f5fb-42ff-82b0-6e0263163d27.86c9f164-921b-4ca4-95dd-68b3c4bb35c2.756c4354-53f9-46c0-b1c3-886e8a2d9bc9 HTTP/2.0", host: "buildbot.lix.systems"
But that line seems to show up somewhat sporadically, not like, when a build is requested or anything | 20:15:14 |
| jade_ | thats oauth stuff | 20:15:41 |
 puck | hrm | 20:15:58 |
| puck | ah, the evals picked up just now | 20:16:09 |
