| 5 Dec 2024 |
 Rick (Mindavi) | In reply to @hexa:lossy.network
so I think the best outcome would be to prune those older than say 3 years? I wouldn't mind that, only recent logs are typically accessed | 15:03:34 |
| Rick (Mindavi) | And people may always rebuild themselves to generate the logs | 15:03:49 |
 vcunat | Aren't they in S3 as well? | 16:36:30 |
| 6 Dec 2024 |
 hexa | systemd.services.hydra-prune-build-logs = {
description = "Clean up old build logs";
startAt = "weekly";
serviceConfig = {
User = "hydra-queue-runner";
Group = "hydra";
ExecStart = lib.concatStringsSep " " [
(lib.getExe pkgs.findutils)
"/var/lib/hydra/build-logs/"
"-type" "f"
"-mtime" "+${toString (3 * 365)}"
"-delete"
];
};
};
| 03:14:04 |
| hexa | In reply to @vcunat:matrix.org
Aren't they in S3 as well?
upload_logs_to_binary_cache = true
| 03:14:31 |
| hexa | only slightly wasteful | 03:14:43 |
| hexa | * systemd.services.hydra-prune-build-logs = {
description = "Clean up old build logs";
startAt = "weekly";
serviceConfig = {
User = "hydra-queue-runner";
Group = "hydra";
ExecStart = lib.concatStringsSep " " [
(lib.getExe pkgs.findutils)
"/var/lib/hydra/build-logs/"
"-type" "f"
"-mtime" "+${toString (3 * 365)}"
"-delete"
];
};
};
deployed to hydra.nixos.org, will run the first time on monday
| 03:15:28 |
| vcunat | It's pretty small compared to the (compressed) NARs, at a glance. But we could have some GC policy there as well. | 06:25:12 |
 das_j | That works but Hydra doesn't fetch them from there. nix logs should do through | 08:47:07 |
| das_j | * That works but Hydra doesn't fetch them from there. nix logs should do though | 08:47:09 |
| vcunat | I do recall some links on Hydra.nixos.org redirecting to URLs that looked like S3 | 08:47:50 |
| vcunat | * I do recall some links on Hydra.nixos.org redirecting to URLs that looked like S3 (and containing logs) | 08:48:02 |
| 8 Dec 2024 |
|  shawn8901 set a profile picture. | 19:21:22 |
| 9 Dec 2024 |
|  karlericsson joined the room. | 07:54:00 |
| 10 Dec 2024 |
 rhelmot | I'm noticing that some part of hydra (hydra-update-gc-roots?) is generating entries in /nix/var/nix/gcroots that are not symlinks or directories but regular empty files with the same name as a /nix/store entry, but nix-collect-garbage isn't respecting them. where is the disconnect here? | 06:21:31 |
 tim | Hey there, did the generated declarative jobset configuration stop working for anyone else in the past ~3 weeks? I have tried to reproduce and this repo does not generate any jobsets besides the initial .jobset.. any ideas how I could debug this further? | 21:36:12 |
| 11 Dec 2024 |
| tim | I had disabled the hydra-notify systemd servce.. 🤦 | 08:32:05 |
|  @dminca:matrix.org left the room. | 14:18:55 |
| 12 Dec 2024 |
| hexa | gcroots prevent garbage collection | 02:23:18 |
| hexa | they come from the number of evaluations kept by the various jobsets | 02:23:33 |
| hexa | gcroot because they are the root of a closure that references one or many store paths | 02:24:13 |
| rhelmot | well... yes, but my question is that the garbage collection is happening anyway | 02:24:16 |
| hexa | how many evals do you keep for your jobset? | 02:24:59 |
| hexa | and does a newer eval maybe replace the older gcroots? | 02:25:12 |
| rhelmot | there have been no new evals | 02:38:47 |
| rhelmot | I have the setting at 3 rn | 02:39:14 |
| rhelmot | the question I originally stated was "is it valid to have a gcroot which is a normal file and not a symlink" | 02:40:34 |
| rhelmot | * the question I originally stated maybe should have been "is it valid to have a gcroot which is a normal file and not a symlink" | 02:40:47 |
|  noisersup changed their profile picture. | 20:04:35 |
| 13 Dec 2024 |
 Christian Theune | hexa: quick update from our end regarding hydra improvements we've planned/queued up (and which mostly ma27 will implement). there's a pr currently waiting to be merged to support globbing in named constituents that also adds logging for memory usage per job. we're currently switching our hydra to zstd compression and will then switch to introducing a hard memory limit. after that i think we'll take a look at moving compression to the workers. I've had one idea to look into making the compression an option on the jobset to allow more gradual changes if this is touched again in the future, but it seems like that might be complicated bordering fragile. and then we'll also take a look at some corner cases where stuck jobs need to be manually cancelled or even killed on the builder itself. as things are currently preparing for the holidays, it looks like most of that will likely happen in the new year. | 06:20:00 |
