| 21 Sep 2021 |
 Mic92 | @ryantm: I rebooted into a recovery system that boots the OS with KVM. Looks like the f2fs filesystem takes ages to do fsck.. | 06:19:45 |
| Mic92 | That might be the reason why takes so long to reboot | 06:20:01 |
| Mic92 | Still doing fsck. | 06:48:01 |
| Mic92 | ryantm: I just did a normal restart again - I suppose non-kvm fsck would be faster than this. I suppose it might take ages until the server come back but I don't see a way around this. | 07:04:48 |
 nix-community-bot | [firing] telegraf_down: build02.nix-community.org:9273: telegraf telegraf exporter from is down.
| 07:26:30 |
| nix-community-bot | [firing] telegraf_down: build02.nix-community.org:9273: telegraf telegraf exporter from is down.
| 11:31:30 |
| nix-community-bot | [resolved] telegraf_down: build02.nix-community.org:9273: telegraf telegraf exporter from is down.
| 15:01:31 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service cachix-watch-store.service.
| 15:06:30 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service cachix-watch-store.service.
| 19:11:30 |
| nix-community-bot | [nix-community/infra] Mic92 pushed to master: marvin-mk2: set primary group - https://github.com/nix-community/infra/commit/e37e1c39ae699c4d0e14ac401aab95c05ab2c277 | 22:59:22 |
| nix-community-bot | [resolved] systemd_service_failed: nix-community-build02 failed to (re)start service cachix-watch-store.service.
| 23:11:30 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build01 failed to (re)start service clone-nixpkgs.service.
| 23:16:31 |
| nix-community-bot | [resolved] systemd_service_failed: nix-community-build01 failed to (re)start service clone-nixpkgs.service.
| 23:21:30 |
| 22 Sep 2021 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-updatescript.service.
| 00:09:30 |
 ryantm | Well it got itself back up enough to start annoying us with the crashes :) | 03:49:07 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-updatescript.service.
| 04:14:30 |
| Mic92 | Yes. it's just that those pauses causes by fsck.f2fs take several hours | 06:07:50 |
| Mic92 | I think I saw those before | 06:08:02 |
| Mic92 | @ryantm: https://wiki.archlinux.org/title/F2FS#Long_running_fsck_delays_boot | 06:08:36 |
| Mic92 | So we have to do this on every kernel upgrade? | 06:09:20 |
| Mic92 | wow | 06:09:33 |
 Sandro | Disable fsck.f2fs | 06:18:02 |
| Sandro | Or use ext4 | 06:18:07 |
| Mic92 | or zfs as we do on other builders | 06:21:31 |
| Mic92 | I don't think disabling fsck.f2fs is an option | 06:21:47 |
| Mic92 | ext4 also has issues with inodes just like f2fs I think | 06:22:24 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-github.service.
[firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-pypi.service.
[firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-updatescript.service.
| 07:19:30 |
| nix-community-bot | [firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-github.service.
[firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-pypi.service.
[firing] systemd_service_failed: nix-community-build02 failed to (re)start service nixpkgs-update-updatescript.service.
| 11:24:30 |
| Mic92 | @ryantm: why do all those services constantly fail lately? | 11:25:42 |
| ryantm | Mic92: By parallelizing the updates, I quadrupled the chances of something crashing. I also fixed some issues where it would sit for a long time without timing out, so it is probably getting to the crashes more quickly. I think it was crashing before just much more slowly. | 14:48:00 |
