| 11 Feb 2025 |
 K900 | @hexa (signing key rotation when) or @John Ericson might have it? | 22:13:16 |
 John Ericson | infinisil: K900 I do not know about this segfault issue | 22:15:18 |
| John Ericson | or this mixed lix nix deployment | 22:15:28 |
| John Ericson | heh | 22:15:33 |
 infinisil | Opened a PR to document this: https://github.com/NixOS/infra/pull/554 | 22:28:25 |
 Robert Hensing (roberth) | same.
A core dump and/or gdb stack trace would be very helpful | 22:48:31 |
 Arian | We have core dump collection enabled at work where we're running 2.19. So we have coredumps for these specific crashes available I think. | 22:57:24 |
| Arian | This is the module we use for that. It's super simplistic. But works: https://github.com/arianvp/nixos-stuff/blob/master/configs/utm/coredump-upload.nix | 23:02:14 |
| Arian | Are we collecting coredumps of the builders already today? Could hook that up to something like sentry or bugsnag or whatever and auto triage this and remove the human in the loop | 23:08:48 |
| Robert Hensing (roberth) | 2.19 is not officially supported anymore, so you may want to update those builders. Nonetheless, outdated coredumps may be valid input for a current bug | 23:30:30 |
 hexa | wow, this seems to have been blown out of proportions | 23:50:39 |
| hexa | this was indeed the case | 23:51:52 |
| hexa | and I might just have forgotten to ever send any backtrace upstream | 23:52:17 |
| hexa | this was a busy time before year's end and the focus was on getting things running without any downtime | 23:52:46 |
| hexa | in fact I took care of that alone 🤷♂️ | 23:52:56 |
| hexa | it is indeed currently a mixed lix on the linux builders, nix on the evaluator, nix on the macs setup | 23:55:08 |
| hexa | collecting that data still requires someone on the other end to evaluate it | 23:56:38 |
| hexa | and I don't think yet another s3 bucket particularly helps, given that we can't seem to be able to deal with the ones we have | 23:57:46 |
| 12 Feb 2025 |
| hexa | finally: I'm not married to that decision or any particular implementation of Nixlang and would like to keep being able to switch things around as needed. There was and is no intent to offend either projects. | 00:03:03 |
| hexa | sorry if this seems like snark, but I really feel like we're not taking the spending problem with AWS too seriously | 00:04:03 |
| hexa | and I personally lack the skills to deal with that, so kudos to whoever can and will | 00:04:21 |
|  @9lore:tchncs.de joined the room. | 00:06:40 |
| Arian | No offense taken. We have it set up that the bucket deletes files older than a month at work. It was mostly just an example. We might not even need to ship the coredumps anywhere. The Hetzner builder machines are long lived as opposed to ephemeral on equinix right? so coredumps should collect in /var/lib/coredumps and the systemd journal anyway | 00:09:51 |
 Winter | yeah, they are long lived | 00:11:24 |
| hexa | and they do | 00:11:59 |
| hexa | the latest coredump is from 2025-01-28 | 00:12:04 |
| hexa | but from a build | 00:12:13 |
| hexa | e.g.
Mon 2025-02-10 07:57:05 UTC 803285 30011 30000 SIGABRT present /build/source/engine/scheduler_test 103.8K
Mon 2025-02-10 07:57:19 UTC 813957 30011 30000 SIGABRT present /build/kyua.fl9Qv2/831/work/kyua.jJYYUN/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814050 30011 30000 SIGABRT present /build/kyua.fl9Qv2/832/work/kyua.s17nJ8/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814181 30011 30000 SIGABRT present /build/kyua.fl9Qv2/833/work/kyua.22SV6O/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814379 30011 30000 SIGABRT present /build/kyua.fl9Qv2/834/work/kyua.J8qa47/1/work/short 44.5K
Mon 2025-02-10 07:57:27 UTC 819457 30011 30000 SIGABRT present /build/kyua.fl9Qv2/843/work/stacktrace_helper 45K
Mon 2025-02-10 07:57:37 UTC 821712 30011 30000 SIGABRT present /build/source/utils/process/isolation_test 97.3K
Mon 2025-02-10 07:57:38 UTC 821829 30011 30000 SIGABRT present /build/source/utils/process/operations_test 95.9K
Mon 2025-02-10 07:57:38 UTC 821855 30011 30000 SIGQUIT present /build/source/utils/process/status_test 94.8K
Mon 2025-02-10 07:58:29 UTC 847236 30001 30000 SIGQUIT present /build/source/atf-c/detail/.libs/lt-process_test 30K
Mon 2025-02-10 07:58:37 UTC 853660 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.2K
Mon 2025-02-10 07:58:37 UTC 854156 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-c_helpers 26.5K
Mon 2025-02-10 07:58:37 UTC 854254 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.7K
| 00:12:49 |
| Arian | Yeh I guess these are dumps from OOM kills or the likes. | 00:13:41 |
 raitobezarius | not necessarily | 00:14:06 |
