| 16 Mar 2026 |
 Vladimír Čunát | Well, five jobs stuck forever in "sending inputs". I'm trying to cancel those now. | 05:54:08 |
| Vladimír Čunát | * Well, five jobs stuck forever in "sending inputs". I'm trying to cancel those now. (and restart them later) | 05:54:21 |
| Vladimír Čunát | I'm looking forward to dropping x86_64-darwin. | 06:05:37 |
| Vladimír Čunát | It's true that the "steps" graph is abnormally low
https://grafana.nixos.org/d/MJw9PcAiz/hydra-jobs?orgId=1&from=now-3h&to=now&refresh=30s&var-machine=$__all&timezone=browser&viewPanel=panel-20 | 06:07:28 |
| Vladimír Čunát | (now that *-linux doesn't have much to do, mostly a few big builds) | 06:08:17 |
 hexa | I don't understand how we're running into out of disk issues so much these last months | 11:15:14 |
| hexa | Redacted or Malformed Event | 11:16:53 |
| hexa | or weeks? 🤔 | 11:17:00 |
| hexa | 
Download image.png | 11:17:24 |
| hexa | 
Download image.png | 11:17:43 |
| hexa | remaining disk space in % | 11:17:47 |
| hexa | * remaining disk space in % < 10 | 11:17:55 |
 Sergei Zimmerman (xokdvium) | Nix bug? | 11:22:38 |
| Sergei Zimmerman (xokdvium) | What's using the disk space? Any crumbs leftover? | 11:23:08 |
| hexa | nix 2.24.10 | 11:25:36 |
| hexa | as boring as it gets | 11:25:41 |
| hexa | nixpkgs-25.11 | 11:25:47 |
| hexa | we even gc twice per hour since https://github.com/NixOS/infra/commit/8b27622bca81efc9b57cf1ff5b61ae34b3373ef3 | 11:26:42 |
| hexa | 228Gi disk on enormous-catfish | 11:29:06 |
| hexa | https://grafana.nixos.org/d/rYdddlPWk/node-exporter-full?orgId=1&from=now-6M&to=now&timezone=browser&var-datasource=default&var-job=node&var-node=enormous-catfish.mac.nixos.org:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B&refresh=1m&viewPanel=panel-152 | 11:29:30 |
| hexa | if it never goes below 90% that means it never gcs more than 23GiB | 11:30:11 |
| hexa | -sh-3.2# df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s1s1 228Gi 11Gi 15Gi 44% 453k 157M 0% /
devfs 204Ki 204Ki 0Bi 100% 706 0 100% /dev
/dev/disk3s6 228Gi 2.0Gi 15Gi 12% 2 157M 0% /System/Volumes/VM
/dev/disk3s4 228Gi 8.2Gi 15Gi 36% 3.7k 157M 0% /System/Volumes/Preboot
/dev/disk3s2 228Gi 65Mi 15Gi 1% 108 157M 0% /System/Volumes/Update
/dev/disk1s2 500Mi 6.0Mi 483Mi 2% 1 4.9M 0% /System/Volumes/xarts
/dev/disk1s1 500Mi 5.7Mi 483Mi 2% 30 4.9M 0% /System/Volumes/iSCPreboot
/dev/disk1s3 500Mi 964Ki 483Mi 1% 39 4.9M 0% /System/Volumes/Hardware
/dev/disk3s3 228Gi 96Gi 15Gi 87% 1.9M 157M 1% /System/Volumes/Data
/dev/disk3s7 228Gi 95Gi 15Gi 87% 5.3M 157M 3% /nix
map auto_home 0Bi 0Bi 0Bi 100% 0 0 - /System/Volumes/Data/home
/dev/disk4 512Mi 12Mi 500Mi 3% 4 4.3G 0% /private/var/run/agenix.d
| 11:30:39 |
| hexa | is the /nix volume constrained in size? 🤔 | 11:31:21 |
| hexa | anyway, the numbers in prometheus and in df don't add up | 11:31:55 |
| hexa | -sh-.2# nix-collect-garbage-d
[...]
32564 store paths deleted, 23203.81 MiB freed
| 11:35:27 |
| hexa | -sh-3.2# df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s1s1 228Gi 11Gi 42Gi 22% 453k 436M 0% /
devfs 203Ki 203Ki 0Bi 100% 704 0 100% /dev
/dev/disk3s6 228Gi 2.0Gi 42Gi 5% 2 436M 0% /System/Volumes/VM
/dev/disk3s4 228Gi 8.2Gi 42Gi 17% 3.7k 436M 0% /System/Volumes/Preboot
/dev/disk3s2 228Gi 65Mi 42Gi 1% 108 436M 0% /System/Volumes/Update
/dev/disk1s2 500Mi 6.0Mi 483Mi 2% 1 4.9M 0% /System/Volumes/xarts
/dev/disk1s1 500Mi 5.7Mi 483Mi 2% 30 4.9M 0% /System/Volumes/iSCPreboot
/dev/disk1s3 500Mi 964Ki 483Mi 1% 39 4.9M 0% /System/Volumes/Hardware
/dev/disk3s3 228Gi 96Gi 42Gi 70% 1.9M 436M 0% /System/Volumes/Data
/dev/disk3s7 228Gi 68Gi 42Gi 62% 2.6M 436M 1% /nix
map auto_home 0Bi 0Bi 0Bi 100% 0 0 - /System/Volumes/Data/home
/dev/disk4 512Mi 12Mi 500Mi 3% 4 4.3G 0% /private/var/run/agenix.d
| 11:35:36 |
| hexa | 
Download image.png | 11:36:19 |
| hexa | which brings it down to 81.8% | 11:36:30 |
| hexa | are we looking at a cow filesystem that incorrectly reports usage data? | 11:37:32 |
 K900 | We have another Pipewire bugfix release | 13:57:36 |
