!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

129 Members
About how to host a very large-scale binary cache and more47 Servers

Load older messages


SenderMessageTime
22 Jun 2024
@hdhog:matrix.hdhog.ruhdhog changed their profile picture.13:23:13
24 Jun 2024
@thefossguy:matrix.orgPratham Patel (you can mention me) joined the room.05:42:56
@thefossguy:matrix.orgPratham Patel (you can mention me)

Hello everyone, I finally have enough SSDs and machines to make sense of a network-wide Nix cache. My use case is as the following:

  1. Fetching binaries from cache.nixos.org and serving on local network.
  2. Any packages that not on cache.nixos.org (i.e. my personal derivations) will be built by the NAS itself by making use of my NixOS config repo and evaluating all [possible] expressions.
  3. Because of 1 and 2, the NAS won't accept any remote build requests nor accept any paths being pushed to it. It will already have everything by the time a local machine requests it.
  4. No authentication because of 3. "Public" serve.

I believe nix-serve should serve me better than attic but I'm not sure. This is mostly a question getting feedback on my approach.

Additionally, I am also thinking if maybe I should netboot all my headless SBCs from this NAS. How should/would that work?

05:54:38
@magic_rb:matrix.redalder.orgmagic_rb
In reply to @thefossguy:matrix.org

Hello everyone, I finally have enough SSDs and machines to make sense of a network-wide Nix cache. My use case is as the following:

  1. Fetching binaries from cache.nixos.org and serving on local network.
  2. Any packages that not on cache.nixos.org (i.e. my personal derivations) will be built by the NAS itself by making use of my NixOS config repo and evaluating all [possible] expressions.
  3. Because of 1 and 2, the NAS won't accept any remote build requests nor accept any paths being pushed to it. It will already have everything by the time a local machine requests it.
  4. No authentication because of 3. "Public" serve.

I believe nix-serve should serve me better than attic but I'm not sure. This is mostly a question getting feedback on my approach.

Additionally, I am also thinking if maybe I should netboot all my headless SBCs from this NAS. How should/would that work?

For the netboot, diregarding how you get the kernel/initrd to the SBC, the rootfs must exist somewhere, your options are tmpfs or nfs. What you could do is only keep /boot on each SBC and then have it mount /nix/store over nfs, and tmpfs on /
06:15:10
@magic_rb:matrix.redalder.orgmagic_rb That would get you almost complete netboot but you dont have to bother with ipxe and such. The wear imposed on the sdcard by keeping /boot there is minimal and you already have to put the ipxe bootloader somewhere so might as well keep boot there 06:16:25
@thefossguy:matrix.orgPratham Patel (you can mention me)ack06:24:56
@janik0:matrix.org@janik0:matrix.org left the room.08:37:05
27 Jun 2024
@hexa:lossy.networkhexatechknowlogick is looking into implementing a nix caching story for gitea15:39:57
@hexa:lossy.networkhexahttps://github.com/go-gitea/gitea/issues/2982415:40:03
@hexa:lossy.networkhexa

This feature request is to add support to Gitea for the HTTP store type.

15:40:19
@hexa:lossy.networkhexawondering if someone can chime in on that issue15:40:24
@hexa:lossy.networkhexa

I've been deep in the docs this past week thinking about how best to implement it

15:40:39
28 Jun 2024
@blitz:chat.x86.lol@blitz:chat.x86.lol left the room.18:09:07
4 Jul 2024
@philiptaron:matrix.org@philiptaron:matrix.org left the room.15:46:20
6 Jul 2024
@sandro:supersandro.deSandro 🐧 joined the room.11:03:12
9 Jul 2024
@sbc64:matrix.orgsbc64 joined the room.16:49:08
11 Jul 2024
@ronnypfannschmidt:matrix.org@ronnypfannschmidt:matrix.orgAre there any docs for setting up and gcing a small personal cache, I have a Homeserver on nix that's available via ssh and id like to use it as cache server for personal needs16:25:40
12 Jul 2024
@samrose:matrix.orgsamrose Ronny: one route is to just use it as a remote build machine https://nix.dev/manual/nix/2.18/advanced-topics/distributed-builds. Nix would automatically cache the builds in the nix store of the machine. you can nix build with --keep-outputs to build up the roots you want to keep. 00:09:00
13 Jul 2024
@ronnypfannschmidt:matrix.org@ronnypfannschmidt:matrix.org
In reply to @samrose:matrix.org
Ronny: one route is to just use it as a remote build machine https://nix.dev/manual/nix/2.18/advanced-topics/distributed-builds. Nix would automatically cache the builds in the nix store of the machine. you can nix build with --keep-outputs to build up the roots you want to keep.
im building on one of my higher powered systems and have the cache on my rpi sytem (that shouldn't build) - i want to work out something where a github action pushes to the cache eventually
10:19:31
14 Jul 2024
@samrose:matrix.orgsamrose Ronny: that is explained in https://nix.dev/manual/nix/2.22/command-ref/new-cli/nix3-copy you just have to figure out how to securely network to your machine. 10:21:20
@ronnypfannschmidt:matrix.org@ronnypfannschmidt:matrix.org
In reply to @samrose:matrix.org
Ronny: that is explained in https://nix.dev/manual/nix/2.22/command-ref/new-cli/nix3-copy you just have to figure out how to securely network to your machine.
this lacks all details on gc-proofing and/or lifecycle management - copied closures dont exactly help with controlling how one reclaims space
10:31:16
@hexa:lossy.networkhexahave you looked at attic so far?11:41:25
@hexa:lossy.networkhexait is a binary cache implementation that uses S3-compatible backends11:42:13
@hexa:lossy.networkhexahttps://github.com/zhaofengli/attic11:42:15
@hexa:lossy.networkhexagc is documented here https://docs.attic.rs/tutorial.html#garbage-collection11:42:34
@hexa:lossy.networkhexai'm using it with a local garage setup, and the same would likely work with minio, or some hosted setup11:44:02
@hexa:lossy.networkhexaif you're interested I can share the config that I run on my local rock5b11:50:09
@ronnypfannschmidt:matrix.org@ronnypfannschmidt:matrix.orgi have taken a look at attic, however as my server has also the largest deploy set, i was hoping to enable using its store directly as a cache instead of creating a extra deployed service (when ssh and filesystem are kinda there)13:14:43
@hexa:lossy.networkhexa you can use harmonia to serve your /nix/store 13:18:33
@hexa:lossy.networkhexa * you can use harmonia to serve your /nix/store 13:18:44

Show newer messages


Back to Room ListRoom Version: 10