| 21 Jan 2022 |
 das_j | In reply to @kenran_:matrix.org
But is it normal that this escalates that quickly w.r.t. disk usage? https://github.com/NixOS/hydra/issues/421 shows a value of 20GB for a much bigger thing. That's from 2016 btw… that's pretty old in Hydra terms ;) | 09:08:42 |
 kenran_ | Ok, thanks, seems like it isn't normal then. I read somewhere that dumps could be quite big if they happen. | 09:08:46 |
| kenran_ | In reply to @janne.hess:helsinki-systems.de
That's from 2016 btw… that's pretty old in Hydra terms ;) Haha yes, but still: they have 100+GB of logs, I have KBs | 09:09:03 |
| kenran_ | I'd expect that everything there would be much bigger numbers. | 09:09:18 |
| das_j | Yeah there's currently no automated method to collect logs 👀 they are only compressed | 09:09:27 |
| kenran_ | That's good to know! Right now the experimental machine I'm on has a separate volume attached that I can increase in size, but /var is still on the primary disk, which is quite small. So that won't scale at all sadly. | 09:14:27 |
| das_j | Currently running du for that on our machine. But you should also worry about your postgres database (no retention here) and your nix store (there is gcroot handling but the store can temporarily grow pretty large) | 09:16:03 |
| das_j | * Currently running du for that on our machineThe Hydra directory is only 20G for us. But you should also worry about your postgres database (no retention here) and your nix store (there is gcroot handling but the store can temporarily grow pretty large) | 09:16:30 |
| kenran_ | (I mean, I could do the same thing I'm doing with /nix/store and mount -o bind,ro ... some directory on the volume "over" /var as well, but that seems error-prune.) | 09:16:46 |
| kenran_ | In reply to @janne.hess:helsinki-systems.de
Currently running du for that on our machineThe Hydra directory is only 20G for us. But you should also worry about your postgres database (no retention here) and your nix store (there is gcroot handling but the store can temporarily grow pretty large) Thanks for looking :) | 09:17:11 |
| kenran_ | So far I tried making the one 13GB-dir that's the reason for the bloat smaller, to no avail. Maybe I'll just try deleting it. | 09:17:40 |
| kenran_ | How would I go about pulling git sources from a custom GitLab instance? | 10:07:25 |
| das_j | Like you would use git | 10:07:47 |
| das_j | * Like you would use git clone | 10:07:54 |
| kenran_ | Oh sorry, didn't mean to send it already, I was still researching and accidentally hit Enter. I mean a private GitLab instance. | 10:08:05 |
| das_j | ah auth? | 10:08:17 |
| das_j | hydra will just call git with HOME=/var/lib/hydra as the hydra user | 10:08:50 |
| das_j | This means that this git will pick up private keys in /var/lib/hydra/.ssh if they are owned by the hydra user | 10:09:09 |
| kenran_ | Yes exactly. I personally pull via SSH, so I guess I'd have to put some SSH key somewhere. | 10:09:25 |
| kenran_ | In reply to @janne.hess:helsinki-systems.de
This means that this git will pick up private keys in /var/lib/hydra/.ssh if they are owned by the hydra user Currently skimming https://github.com/NixOS/hydra/issues/555, which seems to go well with what you wrote. Thanks for clarifying who does the pull. | 10:09:57 |
| kenran_ | I found out I can eval the jobset interactively as said one linked issue further, which makes debugging easier :) | 10:10:14 |
| kenran_ | das_j: Thank you very much, I'm quite some steps further now. I can actually pull, but I'm getting error: access to URI 'https://github.com/NixOS/nixpkgs/archive/b8512aa7f532794ceb8fa31840cbc8e2f1e60432.tar.gz' is forbidden in restricted mode. This looks like our nix expression which uses a pinned version is not allowed to be downloaded, but it just smells like sandboxing and is probably something easy to get working. | 10:54:12 |
| das_j | You need to disable the restricted mode for this | 11:00:08 |
| das_j | https://github.com/NixOS/hydra/pull/888 would do that properly but you can also just patch it out of hydra because the PR is not merged | 11:00:20 |
| das_j | Just add this to postPatch of your Hydra package in the meantime: sed -i 's/evalSettings.restrictEval = true/evalSettings.restrictEval = false/' "$(find -name hydra-eval-jobs.cc)" | 11:01:27 |
| das_j | also cc grahamc (he/him) what about that PR? 👀 | 11:01:35 |
| das_j | * also cc grahamc (he/him) what about the aforementioned PR? 👀 | 11:01:48 |
| kenran_ | Ah, nice, that's your PR :D The PR #981 is merged though, wouldn't setting the option (where though?) suffice? | 11:03:45 |
| das_j | pure eval != restricted eval though :D | 11:04:20 |
| kenran_ | Oh, my bad | 11:04:59 |
