| 16 Mar 2026 |
 hexa | Redacted or Malformed Event | 11:16:53 |
| hexa | or weeks? 🤔 | 11:17:00 |
| hexa | 
Download image.png | 11:17:24 |
| hexa | 
Download image.png | 11:17:43 |
| hexa | remaining disk space in % | 11:17:47 |
| hexa | * remaining disk space in % < 10 | 11:17:55 |
 Sergei Zimmerman (xokdvium) | Nix bug? | 11:22:38 |
| Sergei Zimmerman (xokdvium) | What's using the disk space? Any crumbs leftover? | 11:23:08 |
| hexa | nix 2.24.10 | 11:25:36 |
| hexa | as boring as it gets | 11:25:41 |
| hexa | nixpkgs-25.11 | 11:25:47 |
| hexa | we even gc twice per hour since https://github.com/NixOS/infra/commit/8b27622bca81efc9b57cf1ff5b61ae34b3373ef3 | 11:26:42 |
| hexa | 228Gi disk on enormous-catfish | 11:29:06 |
| hexa | https://grafana.nixos.org/d/rYdddlPWk/node-exporter-full?orgId=1&from=now-6M&to=now&timezone=browser&var-datasource=default&var-job=node&var-node=enormous-catfish.mac.nixos.org:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B&refresh=1m&viewPanel=panel-152 | 11:29:30 |
| hexa | if it never goes below 90% that means it never gcs more than 23GiB | 11:30:11 |
| hexa | -sh-3.2# df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s1s1 228Gi 11Gi 15Gi 44% 453k 157M 0% /
devfs 204Ki 204Ki 0Bi 100% 706 0 100% /dev
/dev/disk3s6 228Gi 2.0Gi 15Gi 12% 2 157M 0% /System/Volumes/VM
/dev/disk3s4 228Gi 8.2Gi 15Gi 36% 3.7k 157M 0% /System/Volumes/Preboot
/dev/disk3s2 228Gi 65Mi 15Gi 1% 108 157M 0% /System/Volumes/Update
/dev/disk1s2 500Mi 6.0Mi 483Mi 2% 1 4.9M 0% /System/Volumes/xarts
/dev/disk1s1 500Mi 5.7Mi 483Mi 2% 30 4.9M 0% /System/Volumes/iSCPreboot
/dev/disk1s3 500Mi 964Ki 483Mi 1% 39 4.9M 0% /System/Volumes/Hardware
/dev/disk3s3 228Gi 96Gi 15Gi 87% 1.9M 157M 1% /System/Volumes/Data
/dev/disk3s7 228Gi 95Gi 15Gi 87% 5.3M 157M 3% /nix
map auto_home 0Bi 0Bi 0Bi 100% 0 0 - /System/Volumes/Data/home
/dev/disk4 512Mi 12Mi 500Mi 3% 4 4.3G 0% /private/var/run/agenix.d
| 11:30:39 |
| hexa | is the /nix volume constrained in size? 🤔 | 11:31:21 |
| hexa | anyway, the numbers in prometheus and in df don't add up | 11:31:55 |
| hexa | -sh-.2# nix-collect-garbage-d
[...]
32564 store paths deleted, 23203.81 MiB freed
| 11:35:27 |
| hexa | -sh-3.2# df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s1s1 228Gi 11Gi 42Gi 22% 453k 436M 0% /
devfs 203Ki 203Ki 0Bi 100% 704 0 100% /dev
/dev/disk3s6 228Gi 2.0Gi 42Gi 5% 2 436M 0% /System/Volumes/VM
/dev/disk3s4 228Gi 8.2Gi 42Gi 17% 3.7k 436M 0% /System/Volumes/Preboot
/dev/disk3s2 228Gi 65Mi 42Gi 1% 108 436M 0% /System/Volumes/Update
/dev/disk1s2 500Mi 6.0Mi 483Mi 2% 1 4.9M 0% /System/Volumes/xarts
/dev/disk1s1 500Mi 5.7Mi 483Mi 2% 30 4.9M 0% /System/Volumes/iSCPreboot
/dev/disk1s3 500Mi 964Ki 483Mi 1% 39 4.9M 0% /System/Volumes/Hardware
/dev/disk3s3 228Gi 96Gi 42Gi 70% 1.9M 436M 0% /System/Volumes/Data
/dev/disk3s7 228Gi 68Gi 42Gi 62% 2.6M 436M 1% /nix
map auto_home 0Bi 0Bi 0Bi 100% 0 0 - /System/Volumes/Data/home
/dev/disk4 512Mi 12Mi 500Mi 3% 4 4.3G 0% /private/var/run/agenix.d
| 11:35:36 |
| hexa | 
Download image.png | 11:36:19 |
| hexa | which brings it down to 81.8% | 11:36:30 |
| hexa | are we looking at a cow filesystem that incorrectly reports usage data? | 11:37:32 |
 K900 | We have another Pipewire bugfix release | 13:57:36 |
| K900 | I'm tempted to send it | 13:57:40 |
| K900 | Because the darwins are stalling everything out anyway | 13:57:46 |
| 17 Mar 2026 |
 emily | APFS is volume-based, like btrfs/ZFS/etc.; the 228 GiB on the Nix store is shared with /System/Volumes/Data, so ~72%? though that's still lower than Prometheus | 00:51:44 |
| hexa | so we get ~60 GiB of good use on those machines? | 00:52:39 |
| hexa | we should get rid off them in that case | 00:52:49 |
| hexa | and replace them with machines that have sensible disk capacity | 00:53:00 |
