| 10 Feb 2025 |
 K900 | I'm going to guess that nej is crashing somehow | 17:28:55 |
| K900 | And stopping mid-line | 17:29:01 |
 hexa | switched to a generation before nej | 17:33:47 |
| K900 | It ded | 17:34:23 |
| hexa | it is a the stopping step 😄 | 17:34:53 |
| hexa | at least the errory is pretty | 17:35:12 |
| hexa | and we're back | 17:37:20 |
| K900 | I've restarted the eval again | 17:46:49 |
| hexa | https://github.com/NixOS/hydra/issues/1437 | 17:47:02 |
| K900 | Well eval hasn't died yet | 18:00:05 |
| K900 | At least | 18:00:07 |
| 11 Feb 2025 |
 infinisil | From here
Lix is the Nix implementation which runs hydra.nixos.org
I'm doubting the accurancy of that:
- https://hydra.nixos.org/ says "using nix-2.24.8" at the bottom
- No mention of Lix in https://github.com/nixos/hydra
- Only this singular use in infra, but I don't think that's used for actual evals, and that line was introduced without any discussion
Am I missing something?
| 22:08:15 |
| K900 | It's used on the builders because cppnix had a weird random segfault bug and that was the easiest thing we could do to get the builds going | 22:09:52 |
| K900 | The coordinator has to be cppnix because that's the only thing our hydra branch works with | 22:10:15 |
| infinisil | Ah so evals are still done using Nix | 22:12:04 |
| infinisil | K900: Can you link me to the segfault issue? | 22:12:26 |
| K900 | Not off the top of my head | 22:12:38 |
| K900 | @hexa (signing key rotation when) or @John Ericson might have it? | 22:13:16 |
 John Ericson | infinisil: K900 I do not know about this segfault issue | 22:15:18 |
| John Ericson | or this mixed lix nix deployment | 22:15:28 |
| John Ericson | heh | 22:15:33 |
| infinisil | Opened a PR to document this: https://github.com/NixOS/infra/pull/554 | 22:28:25 |
 Robert Hensing (roberth) | same.
A core dump and/or gdb stack trace would be very helpful | 22:48:31 |
 Arian | We have core dump collection enabled at work where we're running 2.19. So we have coredumps for these specific crashes available I think. | 22:57:24 |
| Arian | This is the module we use for that. It's super simplistic. But works: https://github.com/arianvp/nixos-stuff/blob/master/configs/utm/coredump-upload.nix | 23:02:14 |
| Arian | Are we collecting coredumps of the builders already today? Could hook that up to something like sentry or bugsnag or whatever and auto triage this and remove the human in the loop | 23:08:48 |
| Robert Hensing (roberth) | 2.19 is not officially supported anymore, so you may want to update those builders. Nonetheless, outdated coredumps may be valid input for a current bug | 23:30:30 |
| hexa | wow, this seems to have been blown out of proportions | 23:50:39 |
| hexa | this was indeed the case | 23:51:52 |
| hexa | and I might just have forgotten to ever send any backtrace upstream | 23:52:17 |
