| 9 Dec 2024 |
|  karlericsson joined the room. | 07:54:00 |
| 10 Dec 2024 |
 rhelmot | I'm noticing that some part of hydra (hydra-update-gc-roots?) is generating entries in /nix/var/nix/gcroots that are not symlinks or directories but regular empty files with the same name as a /nix/store entry, but nix-collect-garbage isn't respecting them. where is the disconnect here? | 06:21:31 |
 tim | Hey there, did the generated declarative jobset configuration stop working for anyone else in the past ~3 weeks? I have tried to reproduce and this repo does not generate any jobsets besides the initial .jobset.. any ideas how I could debug this further? | 21:36:12 |
| 11 Dec 2024 |
| tim | I had disabled the hydra-notify systemd servce.. 🤦 | 08:32:05 |
|  @dminca:matrix.org left the room. | 14:18:55 |
| 12 Dec 2024 |
 hexa | gcroots prevent garbage collection | 02:23:18 |
| hexa | they come from the number of evaluations kept by the various jobsets | 02:23:33 |
| hexa | gcroot because they are the root of a closure that references one or many store paths | 02:24:13 |
| rhelmot | well... yes, but my question is that the garbage collection is happening anyway | 02:24:16 |
| hexa | how many evals do you keep for your jobset? | 02:24:59 |
| hexa | and does a newer eval maybe replace the older gcroots? | 02:25:12 |
| rhelmot | there have been no new evals | 02:38:47 |
| rhelmot | I have the setting at 3 rn | 02:39:14 |
| rhelmot | the question I originally stated was "is it valid to have a gcroot which is a normal file and not a symlink" | 02:40:34 |
| rhelmot | * the question I originally stated maybe should have been "is it valid to have a gcroot which is a normal file and not a symlink" | 02:40:47 |
|  noisersup changed their profile picture. | 20:04:35 |
| 13 Dec 2024 |
 Christian Theune | hexa: quick update from our end regarding hydra improvements we've planned/queued up (and which mostly ma27 will implement). there's a pr currently waiting to be merged to support globbing in named constituents that also adds logging for memory usage per job. we're currently switching our hydra to zstd compression and will then switch to introducing a hard memory limit. after that i think we'll take a look at moving compression to the workers. I've had one idea to look into making the compression an option on the jobset to allow more gradual changes if this is touched again in the future, but it seems like that might be complicated bordering fragile. and then we'll also take a look at some corner cases where stuck jobs need to be manually cancelled or even killed on the builder itself. as things are currently preparing for the holidays, it looks like most of that will likely happen in the new year. | 06:20:00 |
 vcunat | Sounds nice. At hydra.nixos.org we seem to have now avoided the compression bottleneck by brute force (48-core EPYC + hyperthreading). | 07:28:47 |
| Christian Theune | Yeah, our immediate measurement was a reduction of (disk / channel) image compression time going down from 5 minutes to 7s, so that seems a big win. | 08:03:44 |
| Christian Theune | Nevertheless I'm trying to keep an eye on things that happen in a blocking fashion in the queue runner. | 08:04:28 |
| Christian Theune | because those will all be bottlenecks for scaling | 08:04:49 |
| Christian Theune | i'm guessing s3 uploads being in a similar spot | 08:05:05 |
| Christian Theune | if s3 uploads are also in the queue runner blocking things, then I'm wondering whether the uploads could also happen form the workers as long as hydra provides the signature. | 08:06:08 |
| Christian Theune | * if s3 uploads are also in the queue runner blocking things, then I'm wondering whether the uploads could also happen from the workers as long as hydra provides the signature. | 08:06:18 |
| Christian Theune | from a security perspective i understand that we want to keep the signing key on the master | 08:06:29 |
| Christian Theune | s3 upload credentials then aren't really /that/ sensitive compared to that we have to trust the content that the builders generate anyway. | 08:06:49 |
| vcunat | Signing itself is cheap, if you provide the hash to sign. The signer doesn't even need the whole NAR. | 08:07:09 |
| vcunat | * Signing itself is cheap, if you provide the hash to sign. The signer doesn't even need the whole NAR. (in principle) | 08:07:15 |
| Christian Theune | ah, interesting. | 08:07:28 |
| Christian Theune | that would mean we wouldn't even have to transfer the files to the master for that reason. | 08:07:43 |
