| 11 Feb 2025 |
 hexa | this was a busy time before year's end and the focus was on getting things running without any downtime | 23:52:46 |
| hexa | in fact I took care of that alone 🤷♂️ | 23:52:56 |
| hexa | it is indeed currently a mixed lix on the linux builders, nix on the evaluator, nix on the macs setup | 23:55:08 |
| hexa | collecting that data still requires someone on the other end to evaluate it | 23:56:38 |
| hexa | and I don't think yet another s3 bucket particularly helps, given that we can't seem to be able to deal with the ones we have | 23:57:46 |
| 12 Feb 2025 |
| hexa | finally: I'm not married to that decision or any particular implementation of Nixlang and would like to keep being able to switch things around as needed. There was and is no intent to offend either projects. | 00:03:03 |
| hexa | sorry if this seems like snark, but I really feel like we're not taking the spending problem with AWS too seriously | 00:04:03 |
| hexa | and I personally lack the skills to deal with that, so kudos to whoever can and will | 00:04:21 |
|  @9lore:tchncs.de joined the room. | 00:06:40 |
 Arian | No offense taken. We have it set up that the bucket deletes files older than a month at work. It was mostly just an example. We might not even need to ship the coredumps anywhere. The Hetzner builder machines are long lived as opposed to ephemeral on equinix right? so coredumps should collect in /var/lib/coredumps and the systemd journal anyway | 00:09:51 |
 Winter | yeah, they are long lived | 00:11:24 |
| hexa | and they do | 00:11:59 |
| hexa | the latest coredump is from 2025-01-28 | 00:12:04 |
| hexa | but from a build | 00:12:13 |
| hexa | e.g.
Mon 2025-02-10 07:57:05 UTC 803285 30011 30000 SIGABRT present /build/source/engine/scheduler_test 103.8K
Mon 2025-02-10 07:57:19 UTC 813957 30011 30000 SIGABRT present /build/kyua.fl9Qv2/831/work/kyua.jJYYUN/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814050 30011 30000 SIGABRT present /build/kyua.fl9Qv2/832/work/kyua.s17nJ8/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814181 30011 30000 SIGABRT present /build/kyua.fl9Qv2/833/work/kyua.22SV6O/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814379 30011 30000 SIGABRT present /build/kyua.fl9Qv2/834/work/kyua.J8qa47/1/work/short 44.5K
Mon 2025-02-10 07:57:27 UTC 819457 30011 30000 SIGABRT present /build/kyua.fl9Qv2/843/work/stacktrace_helper 45K
Mon 2025-02-10 07:57:37 UTC 821712 30011 30000 SIGABRT present /build/source/utils/process/isolation_test 97.3K
Mon 2025-02-10 07:57:38 UTC 821829 30011 30000 SIGABRT present /build/source/utils/process/operations_test 95.9K
Mon 2025-02-10 07:57:38 UTC 821855 30011 30000 SIGQUIT present /build/source/utils/process/status_test 94.8K
Mon 2025-02-10 07:58:29 UTC 847236 30001 30000 SIGQUIT present /build/source/atf-c/detail/.libs/lt-process_test 30K
Mon 2025-02-10 07:58:37 UTC 853660 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.2K
Mon 2025-02-10 07:58:37 UTC 854156 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-c_helpers 26.5K
Mon 2025-02-10 07:58:37 UTC 854254 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.7K
| 00:12:49 |
| Arian | Yeh I guess these are dumps from OOM kills or the likes. | 00:13:41 |
 raitobezarius | not necessarily | 00:14:06 |
| raitobezarius | testing phase of programs built may trigger SIGABRT on purpose ;-) | 00:14:18 |
| hexa | (gdb) bt full
#0 0x00007ffff7a4c16c in ?? ()
No symbol table info available.
#1 0x00000000004273b0 in ?? ()
No symbol table info available.
#2 0x3693e7cdb4796900 in ?? ()
No symbol table info available.
#3 0x0000000000000006 in ?? ()
No symbol table info available.
#4 0x00007ffff79b1b80 in ?? ()
No symbol table info available.
#5 0x00007fffffffd8c0 in ?? ()
No symbol table info available.
#6 0x0000000000000000 in ?? ()
No symbol table info available.
| 00:15:07 |
| Arian | Ah lol yeh | 00:15:07 |
| hexa | 🙂 | 00:15:08 |
| hexa | but why do those leak into the host, while most other crashes do not? | 00:15:58 |
| hexa | 11 2024-12-24
125 2025-01-24
182 2025-01-25
93 2025-01-26
713 2025-01-27
275 2025-01-28
67 2025-01-29
400 2025-01-30
185 2025-01-31
140 2025-02-01
63 2025-02-02
92 2025-02-03
68 2025-02-04
777 2025-02-05
1173 2025-02-06
429 2025-02-07
16 2025-02-08
255 2025-02-09
724 2025-02-10
1415 2025-02-11
| 00:16:43 |
| hexa | the number of crashes we see per day | 00:17:02 |
| hexa | * the number of crashes we see per day on a single builder | 00:17:08 |
| hexa | most have no corefile | 00:17:17 |
| @9lore:tchncs.de | Hi, just curious https://hydra.nixos.org/build/288757617 is this stuck? Will it timeout at some point? | 00:17:50 |
| Arian | I guess filtering to just the ones we're comm=nix-daemon makes sense | 00:17:52 |
| raitobezarius | In reply to @hexa:lossy.network
but why do those leak into the host, while most other crashes do not? cgroups setup | 00:18:07 |
| raitobezarius | (or I misunderstood your remark) | 00:18:21 |
