| 10 Apr 2023 |
vcunat | With large number of jobs, I suspect the cost (apart from evaluation) is about S3 queries on all output paths across Atlantic (and serialized, I suspect) and then potentially moving lots of data to/from builders. | 14:15:57 |
vcunat | * With large number of jobs, I suspect the cost (apart from evaluation) is about S3 queries on all output paths across Atlantic (and serialized, I suspect) and then potentially moving lots of data to/from builders (some of which could again be relatively far). | 14:16:59 |
vcunat | That's for hydra.nixos.org in particular. Normally you'd probably try putting all the machines close together... | 14:18:36 |
| 11 Apr 2023 |
| @lab-cat:matrix.org joined the room. | 16:06:44 |
| 12 Apr 2023 |
| Yuddite G joined the room. | 09:09:28 |
| 14 Apr 2023 |
aciceri | Have I to do something in particular in order to make remote builder for a different system work on Hydra?
nix = {
buildMachines = [
{
hostName = "rock5b.fleet";
system = "aarch64-linux";
maxJobs = 6;
speedFactor = 2;
supportedFeatures = ["nixos-test" "benchmark" "big-parallel" "kvm"];
mandatoryFeatures = [];
sshKey = "/etc/ssh/ssh_host_ed25519_key";
}
];
distributedBuilds = true;
};
I've this on my machine (x86_64-linux) and I can confirm that I can remote build using nix build <drv>, in Hydra UI "machines" page I see both localhost and ssh://rock5b (aarch64-linux). I've also configured ssh access for the hydra user, so I can access the remote builder using ssh being the hydra user (simply added the ssh keys to /var/lib/hydra/.ssh). Still hydra doesn't build aarch64-linux derivations and gives me "unsupported system type" :(
Am I missing something?
| 07:54:27 |
aciceri | PS: I'm using hydra from a pretty recent nixos-untable updated days/weeks ago | 07:56:36 |
aciceri | Logs don't help, if I manually restart the build from the UI I get this:
Apr 14 09:57:07 mothership hydra-queue-runner[929743]: aborting unsupported build step '/nix/store/cf0kfmf12cqrcvjjsp48zbzp20nw1q5q-treefmt.toml.drv' (type 'aarch64-linux')
Apr 14 09:57:07 mothership hydra-queue-runner[929743]: marking build 371 as failed
| 07:58:10 |
vcunat | I don't really know these things, but I'd look if this appears right: /admin/machines | 08:01:05 |
vcunat | * I don't really know these things, but I'd look if this appears right: http://*/admin/machines | 08:01:20 |
aciceri | Checked, it appears and with same fields I've set in nix.buildMachines | 08:02:37 |
@linus:schreibt.jetzt | zrsk: does sudo -u hydra-queue-runner ssh rock5b.fleet work? | 08:03:22 |
@linus:schreibt.jetzt | note that the builds are triggered by the hydra-queue-runner user, not hydra | 08:03:49 |
aciceri | In reply to @linus:schreibt.jetzt zrsk: does sudo -u hydra-queue-runner ssh rock5b.fleet work? nope, has to be the hydra-queue-runner to be able to access the remote builder? | 08:04:08 |
@linus:schreibt.jetzt | yep | 08:04:15 |
@linus:schreibt.jetzt | hydra doesn't need it, but hydra-queue-runner does | 08:04:31 |
aciceri | Instead for fetching private repos I need to give ssh access to repos to hydra, right? | 08:05:20 |
@linus:schreibt.jetzt | yes | 08:05:26 |
aciceri | I'm trying btw :) | 08:05:34 |
@linus:schreibt.jetzt | if they're fetched at build time it gets tricky, but if they're flake inputs or jobset inputs it should be fine | 08:06:04 |
aciceri | if they're fetched at build times can't I simply add them to allowed-uris? | 08:06:52 |
aciceri | and if I need the remote builder during an IFD does it work? | 08:07:17 |
@linus:schreibt.jetzt | if they're fetched at build time it runs on the build machines, so those need to have keys and the keys somehow need to end up in the build sandbox as well | 08:07:54 |
@linus:schreibt.jetzt | IFD is bad news, I'd suggest doing your best to avoid it | 08:08:36 |
@linus:schreibt.jetzt | I think it will work if you have it in nix.buildMachines and the evaluator uses the daemon to access the store, but YMMV hard on IFD | 08:09:36 |
aciceri | I know they are problematic even without Hydra | 08:11:43 |
@linus:schreibt.jetzt | even more so with hydra :) | 08:11:52 |
aciceri | however it doesn't work :( | 08:12:02 |
aciceri | are you sure that I had to be able to run sudo -u hydra-queue-runner ssh rock5b.fleet? | 08:12:18 |
aciceri | I mean, I don't even have the hydra-queue-runner on the rock5g | 08:12:35 |