!UKDpaKNNsBpOPfLWfX:zhaofeng.li

Colmena

292 Members
A simple, stateless NixOS deployment tool - https://github.com/zhaofengli/colmena101 Servers

Load older messages

SenderMessageTime
3 Dec 2023
@sivertism:matrix.org@sivertism:matrix.org
In reply to @sivertism:matrix.org
So if this works on the surface, I've got all three computers deployed from Colmena. Very cool stuff!

meh, new errror. This one feels like it's got more to do with nixos-hardware than Colmena. Seems like the Colmena side of things is working fine though.

[nix-shell:~/nixos-home]$ colmena apply --on surface
[INFO ] Using flake: git+file:///home/nixos/nixos-home
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 3 hosts.
        ❌ 14s Failed: Child process exited with error code: 1
surface ✅ 10s Evaluated surface
surface ✅ 0s Built "/nix/store/s9mlmd8b6lxy2b5m5wwy099d924y5s0w-nixos-system-surface-24.05pre-git"
surface ✅ 2s Pushed system closure
surface ❌ 0s Activation failed: Child process exited with error code: 1
[ERROR] Failed to deploy to surface - Last 15 lines of logs:
[ERROR]  created)
[ERROR]    state) Running
[ERROR]   stderr)
[ERROR]   stderr) We trust you have received the usual lecture from the local System
[ERROR]   stderr) Administrator. It usually boils down to these three things:
[ERROR]   stderr)
[ERROR]   stderr)     #1) Respect the privacy of others.
[ERROR]   stderr)     #2) Think before you type.
[ERROR]   stderr)     #3) With great power comes great responsibility.
[ERROR]   stderr)
[ERROR]   stderr) For security reasons, the password you type will not be visible.
[ERROR]   stderr)
[ERROR]   stderr) sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
[ERROR]   stderr) sudo: a password is required
[ERROR]  failure) Child process exited with error code: 1
[ERROR] Failed to complete requested operation - Last 1 lines of logs:
[ERROR]  failure) Child process exited with error code: 1
[ERROR] -----
[ERROR] Operation failed with error: Child process exited with error code: 1
Hint: Backtrace available - Use `RUST_BACKTRACE=1` environment variable to display a backtrace
16:07:59
@ibizaman:matrix.orgibizaman Sivert: you need to have password less sudo for the nixos user 16:12:25
@ibizaman:matrix.orgibizaman 
  security.sudo.extraRules = [
    { users = [ "nixos" ];
      commands = [
        { command = "ALL";
          options = [ "NOPASSWD" ];
        }
      ];
    }
  ];
16:13:53
@sivertism:matrix.org@sivertism:matrix.org

ah, you're right! I thought I had done that, but I hadn't.

Got a spicier error message this time though

[nixos@desktop:~/nixos-home]$ colmena apply --on surface
warning: Git tree '/home/nixos/nixos-home' is dirty
[INFO ] Using flake: git+file:///home/nixos/nixos-home
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 3 hosts.
        ❌ 25s Failed: Child process exited with error code: 4
surface ✅ 11s Evaluated surface
surface ✅ 0s Built "/nix/store/nzpx3si3lnw205sdd4vcvkh74b8blvx4-nixos-system-surface-24.05pre-git"
surface ✅ 3s Pushed system closure
surface ❌ 11s Activation failed: Child process exited with error code: 4
[ERROR] Failed to deploy to surface - Last 20 lines of logs:
[ERROR]   stderr) restarting the following units: dev-hugepages.mount, nix-daemon.service, polkit.service, sshd.service, systemd-journald.service
[ERROR]   stderr) starting the following units: ModemManager.service, NetworkManager-wait-online.service, NetworkManager.service, accounts-daemon.service, alsa-store.service, audit.service, avahi-daemon.socket, bluetooth.service, colord.service, cpufreq.service, cups-browsed.service, cups.socket, kmod-static-nodes.service, logrotate-checkconf.service, mount-pstore.service, network-local-commands.service, network-setup.service, nscd.service, power-profiles-daemon.service, resolvconf.service, rtkit-daemon.service, systemd-machined.service, systemd-modules-load.service, systemd-oomd.socket, systemd-sysctl.service, systemd-timesyncd.service, systemd-tmpfiles-setup-dev.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, udisks2.service, upower.service, wpa_supplicant.service
[ERROR]   stderr) the following new units were started: NetworkManager-dispatcher.service, fstrim.timer, systemd-hostnamed.service, systemd-tmpfiles-setup-dev-early.service, systemd-vconsole-setup.service
[ERROR]   stderr) warning: the following units failed: iptsd.service
[ERROR]   stderr)
[ERROR]   stderr) × iptsd.service - IPTSD
[ERROR]   stderr)      Loaded: loaded (/etc/systemd/system/iptsd.service; enabled; preset: enabled)
[ERROR]   stderr)      Active: failed (Result: exit-code) since Sun 2023-12-03 17:17:38 CET; 3s ago
[ERROR]   stderr)    Duration: 295ms
[ERROR]   stderr)     Process: 3490 ExecStart=/nix/store/a2vzhri5pmfs83fpwcngm4inwk2ww7yj-unit-script-iptsd-start/bin/iptsd-start (code=exited, status=106)
[ERROR]   stderr)    Main PID: 3490 (code=exited, status=106)
[ERROR]   stderr)          IP: 0B in, 0B out
[ERROR]   stderr)         CPU: 99ms
[ERROR]   stderr)
[ERROR]   stderr) Dec 03 17:17:37 surface systemd[1]: Started IPTSD.
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: DEVICE is required
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: Run with --help for more information.
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Main process exited, code=exited, status=106/n/a
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Failed with result 'exit-code'.
[ERROR]  failure) Child process exited with error code: 4
[ERROR] Failed to complete requested operation - Last 1 lines of logs:
[ERROR]  failure) Child process exited with error code: 4
[ERROR] -----
[ERROR] Operation failed with error: Child process exited with error code: 4
Hint: Backtrace available - Use `RUST_BACKTRACE=1` environment variable to display a backtrace
16:19:17
@sivertism:matrix.org@sivertism:matrix.org
In reply to @sivertism:matrix.org

ah, you're right! I thought I had done that, but I hadn't.

Got a spicier error message this time though

[nixos@desktop:~/nixos-home]$ colmena apply --on surface
warning: Git tree '/home/nixos/nixos-home' is dirty
[INFO ] Using flake: git+file:///home/nixos/nixos-home
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 3 hosts.
        ❌ 25s Failed: Child process exited with error code: 4
surface ✅ 11s Evaluated surface
surface ✅ 0s Built "/nix/store/nzpx3si3lnw205sdd4vcvkh74b8blvx4-nixos-system-surface-24.05pre-git"
surface ✅ 3s Pushed system closure
surface ❌ 11s Activation failed: Child process exited with error code: 4
[ERROR] Failed to deploy to surface - Last 20 lines of logs:
[ERROR]   stderr) restarting the following units: dev-hugepages.mount, nix-daemon.service, polkit.service, sshd.service, systemd-journald.service
[ERROR]   stderr) starting the following units: ModemManager.service, NetworkManager-wait-online.service, NetworkManager.service, accounts-daemon.service, alsa-store.service, audit.service, avahi-daemon.socket, bluetooth.service, colord.service, cpufreq.service, cups-browsed.service, cups.socket, kmod-static-nodes.service, logrotate-checkconf.service, mount-pstore.service, network-local-commands.service, network-setup.service, nscd.service, power-profiles-daemon.service, resolvconf.service, rtkit-daemon.service, systemd-machined.service, systemd-modules-load.service, systemd-oomd.socket, systemd-sysctl.service, systemd-timesyncd.service, systemd-tmpfiles-setup-dev.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, udisks2.service, upower.service, wpa_supplicant.service
[ERROR]   stderr) the following new units were started: NetworkManager-dispatcher.service, fstrim.timer, systemd-hostnamed.service, systemd-tmpfiles-setup-dev-early.service, systemd-vconsole-setup.service
[ERROR]   stderr) warning: the following units failed: iptsd.service
[ERROR]   stderr)
[ERROR]   stderr) × iptsd.service - IPTSD
[ERROR]   stderr)      Loaded: loaded (/etc/systemd/system/iptsd.service; enabled; preset: enabled)
[ERROR]   stderr)      Active: failed (Result: exit-code) since Sun 2023-12-03 17:17:38 CET; 3s ago
[ERROR]   stderr)    Duration: 295ms
[ERROR]   stderr)     Process: 3490 ExecStart=/nix/store/a2vzhri5pmfs83fpwcngm4inwk2ww7yj-unit-script-iptsd-start/bin/iptsd-start (code=exited, status=106)
[ERROR]   stderr)    Main PID: 3490 (code=exited, status=106)
[ERROR]   stderr)          IP: 0B in, 0B out
[ERROR]   stderr)         CPU: 99ms
[ERROR]   stderr)
[ERROR]   stderr) Dec 03 17:17:37 surface systemd[1]: Started IPTSD.
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: DEVICE is required
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: Run with --help for more information.
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Main process exited, code=exited, status=106/n/a
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Failed with result 'exit-code'.
[ERROR]  failure) Child process exited with error code: 4
[ERROR] Failed to complete requested operation - Last 1 lines of logs:
[ERROR]  failure) Child process exited with error code: 4
[ERROR] -----
[ERROR] Operation failed with error: Child process exited with error code: 4
Hint: Backtrace available - Use `RUST_BACKTRACE=1` environment variable to display a backtrace

Just needed a reboot. Works like a charm now. Touch and everything 🥳

Thanks a lot for the help ibizaman !

16:24:47
@ibizaman:matrix.orgibizaman 
The error is weird indeed. Glad we made it work!
17:20:18
@ibizaman:matrix.orgibizaman *

The error is weird indeed. Glad we made it work!

17:20:39
@zhaofeng:zhaofeng.liZhaofeng Li
In reply to @sivertism:matrix.org

ah, you're right! I thought I had done that, but I hadn't.

Got a spicier error message this time though

[nixos@desktop:~/nixos-home]$ colmena apply --on surface
warning: Git tree '/home/nixos/nixos-home' is dirty
[INFO ] Using flake: git+file:///home/nixos/nixos-home
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 3 hosts.
        ❌ 25s Failed: Child process exited with error code: 4
surface ✅ 11s Evaluated surface
surface ✅ 0s Built "/nix/store/nzpx3si3lnw205sdd4vcvkh74b8blvx4-nixos-system-surface-24.05pre-git"
surface ✅ 3s Pushed system closure
surface ❌ 11s Activation failed: Child process exited with error code: 4
[ERROR] Failed to deploy to surface - Last 20 lines of logs:
[ERROR]   stderr) restarting the following units: dev-hugepages.mount, nix-daemon.service, polkit.service, sshd.service, systemd-journald.service
[ERROR]   stderr) starting the following units: ModemManager.service, NetworkManager-wait-online.service, NetworkManager.service, accounts-daemon.service, alsa-store.service, audit.service, avahi-daemon.socket, bluetooth.service, colord.service, cpufreq.service, cups-browsed.service, cups.socket, kmod-static-nodes.service, logrotate-checkconf.service, mount-pstore.service, network-local-commands.service, network-setup.service, nscd.service, power-profiles-daemon.service, resolvconf.service, rtkit-daemon.service, systemd-machined.service, systemd-modules-load.service, systemd-oomd.socket, systemd-sysctl.service, systemd-timesyncd.service, systemd-tmpfiles-setup-dev.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, udisks2.service, upower.service, wpa_supplicant.service
[ERROR]   stderr) the following new units were started: NetworkManager-dispatcher.service, fstrim.timer, systemd-hostnamed.service, systemd-tmpfiles-setup-dev-early.service, systemd-vconsole-setup.service
[ERROR]   stderr) warning: the following units failed: iptsd.service
[ERROR]   stderr)
[ERROR]   stderr) × iptsd.service - IPTSD
[ERROR]   stderr)      Loaded: loaded (/etc/systemd/system/iptsd.service; enabled; preset: enabled)
[ERROR]   stderr)      Active: failed (Result: exit-code) since Sun 2023-12-03 17:17:38 CET; 3s ago
[ERROR]   stderr)    Duration: 295ms
[ERROR]   stderr)     Process: 3490 ExecStart=/nix/store/a2vzhri5pmfs83fpwcngm4inwk2ww7yj-unit-script-iptsd-start/bin/iptsd-start (code=exited, status=106)
[ERROR]   stderr)    Main PID: 3490 (code=exited, status=106)
[ERROR]   stderr)          IP: 0B in, 0B out
[ERROR]   stderr)         CPU: 99ms
[ERROR]   stderr)
[ERROR]   stderr) Dec 03 17:17:37 surface systemd[1]: Started IPTSD.
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: DEVICE is required
[ERROR]   stderr) Dec 03 17:17:38 surface iptsd-start[3619]: Run with --help for more information.
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Main process exited, code=exited, status=106/n/a
[ERROR]   stderr) Dec 03 17:17:38 surface systemd[1]: iptsd.service: Failed with result 'exit-code'.
[ERROR]  failure) Child process exited with error code: 4
[ERROR] Failed to complete requested operation - Last 1 lines of logs:
[ERROR]  failure) Child process exited with error code: 4
[ERROR] -----
[ERROR] Operation failed with error: Child process exited with error code: 4
Hint: Backtrace available - Use `RUST_BACKTRACE=1` environment variable to display a backtrace

Looks like the iptsd setup in nixos-hardware does script = "iptsd $(iptsd-find-hidraw)";

https://github.com/NixOS/nixos-hardware/blob/a89745edd5f657e2e5be5ed1bea86725ca78d92e/microsoft/surface/common/ipts/default.nix#L42

17:31:50
@zhaofeng:zhaofeng.liZhaofeng Li If you don't have the driver, then iptsd-find-hidraw presumably returns nothing 17:32:15
@zhaofeng:zhaofeng.liZhaofeng Lihence "DEVICE is required"17:32:34
@sivertism:matrix.org@sivertism:matrix.orgAh, that makes sense. Thanks!18:00:45
4 Dec 2023
@ibizaman:matrix.orgibizaman Sivert: I felt compelled to make a blog post out of it, I hope you don't mind https://blog.tiserbox.com/posts/2023-12-03-2-deploy-to-nix-os-without-root-user.html 04:15:19
@sivertism:matrix.org@sivertism:matrix.orgNot at all, I think it's a good idea to have all the steps in one place 👍️05:23:42
5 Dec 2023
@federicodschonborn:matrix.org@federicodschonborn:matrix.org changed their profile picture.00:38:02
7 Dec 2023
@mutantmell:helveticastandard.com@mutantmell:helveticastandard.com left the room.03:55:45
11 Dec 2023
@ar:is-a.catari ❄ colmena, when used in a flake, requires that one sets colmena.meta.nixpkgs, and that definition typically takes the system argument. I have a flake with configs for aarch64-linux and x86_64-linux, generating the nixosConfigurations entries using nixpkgs.lib.nixosSystem that also takes the system argument. now, if I don't additionally explicitly set nixpkgs.system for all the hosts, building the configs using nixos-rebuild or nix build or whatever works correctly, but colmena appears to set the system for all the hosts to whatever is passed as system argument to colmena.meta.nixpkgs (colmena eval -E '{ nodes, ... }: nodes.scylla.config.nixpkgs.system' returns wrong value).
Is there a workaround for that, that wouldn't involve setting nixpkgs.system explicitly, or adding nodeNixpkgs = builtins.mapAttrs (name: value: value.pkgs) conf; (this duplicates contents of nixpkgs.overlays which can cause other issues)?
14:34:17
@jasom:matrix.orgJasom joined the room.23:20:53
@jasom:matrix.orgJasomColmena is stateless; nixops carries a small amount of state (e.g. system.stateVersion, ssh keys). I happen to like having that state managed for me, and it shouldn't be too hard to make an external tool to do this. Has someone written a tool to do so with Colmena, or should I write one?23:50:17
12 Dec 2023
@whentze:matrix.orgWanja Hentzenot to my knowledge01:51:27
@whentze:matrix.orgWanja Hentzesounds lime a good thing to have available though!01:51:36
@whentze:matrix.orgWanja Hentze* sounds like a good thing to have available though!01:51:56
@ibizaman:matrix.orgibizaman Jasom can you elaborate on what this state management or I guess the lack of thereof means in practice? I don’t remember ever having to deal with that.  01:58:42
@jasom:matrix.orgJasom
In reply to @ibizaman:matrix.org
Jasom can you elaborate on what this state management or I guess the lack of thereof means in practice? I don’t remember ever having to deal with that. 
The system.stateVersion is the biggest one; if it goes in your configuration, then if you wipe a VM (or machine for that matter) and reprovision, it will be wrong if you've ever upgraded nixos to a new release. If it's not in your configuration then it will be wrong as soon as you upgrade to a new release. Nixops kept track of this (and some other state). IIRC it gave each machine a unique ID, checked that it hadn't changed and stored the information in an SQLite database in ~/.nixops. It also performs ssh key management for you, generating (and storing) a private key for each machine and adding the public key to the authorized_keys on the remote. I suspect there's other things it does as well, but I haven't dug too deeply. 		04:11:05
@jasom:matrix.orgJasom To clarify, I think Colmena not doing these things is the right choice, since e.g. auto-provisioning new VMs in the cloud is out-of-scope and silently storing unencrypted keys that allow root access to servers in the user's home directory is a ... suboptimal default. However authentication still needs to happen, so I'm pondering a tool that would work along-side Colmena to do some of these things. 04:15:31
@ibizaman:matrix.orgibizamanI see, indeed I needed to do some manual work to generate a ssh key and add it to the target.04:17:20
@ibizaman:matrix.orgibizamanI remember reading about stateVersion, it makes sure you're introducing only backwards compatible changes and you can update it when you went through all the manual steps outlined in the release notes.04:18:17
16 Dec 2023
@dstengele:pango.placeDennis Stengele changed their profile picture.18:17:13
17 Dec 2023
@mjolnir:nixos.orgNixOS Moderation Bot banned @blaggacao:matrix.org@blaggacao:matrix.org (true).17:51:29
19 Dec 2023
@areskul:matrix.org@areskul:matrix.org joined the room.04:59:03
@areskul:matrix.org@areskul:matrix.org Jasom: I need a tool like this too! I am open to contribute! 05:04:04

Show newer messages

Back to Room ListRoom Version: 6