| 12 Feb 2025 |
 Arian | No offense taken. We have it set up that the bucket deletes files older than a month at work. It was mostly just an example. We might not even need to ship the coredumps anywhere. The Hetzner builder machines are long lived as opposed to ephemeral on equinix right? so coredumps should collect in /var/lib/coredumps and the systemd journal anyway | 00:09:51 |
 Winter | yeah, they are long lived | 00:11:24 |
 hexa | and they do | 00:11:59 |
| hexa | the latest coredump is from 2025-01-28 | 00:12:04 |
| hexa | but from a build | 00:12:13 |
| hexa | e.g.
Mon 2025-02-10 07:57:05 UTC 803285 30011 30000 SIGABRT present /build/source/engine/scheduler_test 103.8K
Mon 2025-02-10 07:57:19 UTC 813957 30011 30000 SIGABRT present /build/kyua.fl9Qv2/831/work/kyua.jJYYUN/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814050 30011 30000 SIGABRT present /build/kyua.fl9Qv2/832/work/kyua.s17nJ8/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814181 30011 30000 SIGABRT present /build/kyua.fl9Qv2/833/work/kyua.22SV6O/1/work/short 44.4K
Mon 2025-02-10 07:57:20 UTC 814379 30011 30000 SIGABRT present /build/kyua.fl9Qv2/834/work/kyua.J8qa47/1/work/short 44.5K
Mon 2025-02-10 07:57:27 UTC 819457 30011 30000 SIGABRT present /build/kyua.fl9Qv2/843/work/stacktrace_helper 45K
Mon 2025-02-10 07:57:37 UTC 821712 30011 30000 SIGABRT present /build/source/utils/process/isolation_test 97.3K
Mon 2025-02-10 07:57:38 UTC 821829 30011 30000 SIGABRT present /build/source/utils/process/operations_test 95.9K
Mon 2025-02-10 07:57:38 UTC 821855 30011 30000 SIGQUIT present /build/source/utils/process/status_test 94.8K
Mon 2025-02-10 07:58:29 UTC 847236 30001 30000 SIGQUIT present /build/source/atf-c/detail/.libs/lt-process_test 30K
Mon 2025-02-10 07:58:37 UTC 853660 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.2K
Mon 2025-02-10 07:58:37 UTC 854156 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-c_helpers 26.5K
Mon 2025-02-10 07:58:37 UTC 854254 30001 30000 SIGABRT present /build/source/test-programs/.libs/lt-cpp_helpers 56.7K
| 00:12:49 |
| Arian | Yeh I guess these are dumps from OOM kills or the likes. | 00:13:41 |
 raitobezarius | not necessarily | 00:14:06 |
| raitobezarius | testing phase of programs built may trigger SIGABRT on purpose ;-) | 00:14:18 |
| hexa | (gdb) bt full
#0 0x00007ffff7a4c16c in ?? ()
No symbol table info available.
#1 0x00000000004273b0 in ?? ()
No symbol table info available.
#2 0x3693e7cdb4796900 in ?? ()
No symbol table info available.
#3 0x0000000000000006 in ?? ()
No symbol table info available.
#4 0x00007ffff79b1b80 in ?? ()
No symbol table info available.
#5 0x00007fffffffd8c0 in ?? ()
No symbol table info available.
#6 0x0000000000000000 in ?? ()
No symbol table info available.
| 00:15:07 |
| Arian | Ah lol yeh | 00:15:07 |
| hexa | 🙂 | 00:15:08 |
| hexa | but why do those leak into the host, while most other crashes do not? | 00:15:58 |
| hexa | 11 2024-12-24
125 2025-01-24
182 2025-01-25
93 2025-01-26
713 2025-01-27
275 2025-01-28
67 2025-01-29
400 2025-01-30
185 2025-01-31
140 2025-02-01
63 2025-02-02
92 2025-02-03
68 2025-02-04
777 2025-02-05
1173 2025-02-06
429 2025-02-07
16 2025-02-08
255 2025-02-09
724 2025-02-10
1415 2025-02-11
| 00:16:43 |
| hexa | the number of crashes we see per day | 00:17:02 |
| hexa | * the number of crashes we see per day on a single builder | 00:17:08 |
| hexa | most have no corefile | 00:17:17 |
 @9lore:tchncs.de | Hi, just curious https://hydra.nixos.org/build/288757617 is this stuck? Will it timeout at some point? | 00:17:50 |
| Arian | I guess filtering to just the ones we're comm=nix-daemon makes sense | 00:17:52 |
| raitobezarius | In reply to @hexa:lossy.network
but why do those leak into the host, while most other crashes do not? cgroups setup | 00:18:07 |
| raitobezarius | (or I misunderstood your remark) | 00:18:21 |
| hexa | likely after 2 hours of silence | 00:18:29 |
| hexa | raitobezarius: the question was: why do some few have corefiles, but most do not, when we're all running them in cgroups | 00:19:13 |
| hexa | * raitobezarius: the question was: why do some few have corefiles, but most do not, when we're all running them in the build sandbox | 00:19:23 |
| raitobezarius | corefiles are pruned | 00:19:28 |
| hexa | yes, but even new crashes have none 😄 | 00:19:39 |
| Arian | Systemd deletes them after a while | 00:19:41 |
| raitobezarius | maybe the vacuum algorithm is weird | 00:19:53 |
| raitobezarius | also corefiles cannot always be obtained | 00:20:08 |
| raitobezarius | some crashes simply does not come with corefiles | 00:20:12 |
