| 19 Feb 2026 |
 Jeremy Fleischman (jfly) | (The benefit of doing it this way is we have 30 days to figure out if we did something wrong) | 19:47:16 |
 hexa (signing key rotation when) | # zpool status
pool: zroot
state: DEGRADED
status: One or more devices have been removed.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 00:03:45 with 0 errors on Sun Feb 1 02:17:51 2026
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
nvme-eui.002538b931005446-part3 ONLINE 0 0 0
nvme-eui.002538b931005449-part3 REMOVED 0 0 0
errors: No known data errors
| 20:12:45 |
| hexa (signing key rotation when) | on pluto | 20:12:47 |
| hexa (signing key rotation when) | collected debug logs, deconfigured the faulty disk and rebooted | 20:25:33 |
| hexa (signing key rotation when) | sent a ticket for a replacement disk | 20:29:58 |
| hexa (signing key rotation when) | the machine is back up and the disk was not enumerated | 20:30:17 |
| hexa (signing key rotation when) | we got a new disk and resilvering is in progress | 22:20:08 |
| hexa (signing key rotation when) | will finish in a minute or so | 22:21:43 |
| hexa (signing key rotation when) | and a final reboot to check if everything comes up correctly | 22:29:52 |
| hexa (signing key rotation when) | ok, looks good to me | 22:33:36 |
| 21 Feb 2026 |
|  h7x4 changed their profile picture. | 19:26:51 |
| h7x4 changed their profile picture. | 19:30:19 |
| h7x4 changed their profile picture. | 19:33:47 |
| 22 Feb 2026 |
|  Inlet joined the room. | 02:54:23 |
|  mei 🌒& changed their profile picture. | 22:53:19 |
| 23 Feb 2026 |
| hexa (signing key rotation when) | disk failure on the new disk in pluto | 01:24:35 |
| hexa (signing key rotation when) | I'd say the slot is bad | 01:26:37 |
| hexa (signing key rotation when) | if I had to guess it might be the voltage regulation | 01:27:00 |
| hexa (signing key rotation when) | the temperatures were fine | 01:27:08 |
| hexa (signing key rotation when) | two failing nvmes within 3 days or so is worrying | 01:27:33 |
| hexa (signing key rotation when) | and my conclusion is that we should replace the machine | 01:27:40 |
| hexa (signing key rotation when) | https://www.hetzner.com/pressroom/statement-setup-fees-adjustment/ | 01:27:48 |
| hexa (signing key rotation when) | so now I found this statement after I saw the new setup feees | 01:28:05 |
| hexa (signing key rotation when) | * so now I found this statement after I saw the new setup fees | 01:28:07 |
| hexa (signing key rotation when) | 
Download image.png | 01:29:05 |
| hexa (signing key rotation when) | 
Download image.png | 01:29:15 |
 Winter | 0eur on AX41 lol? | 01:30:57 |
|  Philip joined the room. | 01:35:41 |
| hexa (signing key rotation when) | yeah, that might be the alternative if they can't/won't replace the anything but the disk | 01:48:27 |
| hexa (signing key rotation when) | * yeah, that might be the alternative if they can't/won't replace anything but the disk | 01:48:33 |
