 | DevOS | 37 Members |
| Seeking help and geeking out together on https://github.com/divnix/devos & https://github.com/divnix/digga | 10 Servers |
| DevOS | 37 Members |
| Seeking help and geeking out together on https://github.com/divnix/devos & https://github.com/divnix/digga | 10 Servers |
| Sender | Message | Time |
|---|---|---|
| 30 Jul 2021 | ||
 d4hines | Wow, haven't heard of nvfetcher - so confused 😆 Maybe you guys could give me your recommended approach on a concrete problem. I work on the Tezos blockchain, which is a large OCaml project. So naturally we have lots of OCaml packages, but also some rust and c dependencies. There's probably about 100-150 dependencies total. If this were Javascript, I would just list them all, and npm would go fetch them, stick them in node_modules, and tadah! webpack or whatever knows where everything is (I'm new to native development if you can't tell). My goal is to get back to something like that if possible. Most of the packages are in nixpkgs already, but some are missing, and some of the versions aren't right, so one of my coworkers has this massive overlay he uses to build it, but it seems like a big mess. How would you guys tackle something like this? One thought I had was to fork nixpkgs and pass it as flake input. Maybe even par it down to just the packages we need. But this also seems like a lot of work. Any direction is greatly appreciated! | 15:40:35 |
 @timdeh:matrix.org | Have you heard of the esy package manager for ocaml? Perhaps this could help you guys out a bit? I was actually wanting to make a wrapper for it at some point or something, so I could eventually write a package for oni2. ATM though, I'm not sure how well this would integrate with Nix. Maybe we could team up on an esy2nix side project 😅 | 15:43:55 |
| d4hines | Actually, I work with the esy maintainers! | 15:44:24 |
| @timdeh:matrix.org | Nice, so are you guys using it in your build chain? | 15:44:50 |
| d4hines | Yes and no. Tezos proper does not use it. However, my company, Marigold, has other projects that integrate with Tezos and we use esy for those. | 15:45:38 |
| d4hines | Let's assume there was an esy2nix. What exactly does this do for us? I guess it means that instead of having to update every OCaml package in nixpkgs, Nix now knows how to fetch and build any OCaml package living in the OPAM repository, right (a lot like package.json/npm, right?) | 15:48:58 |
| d4hines | So we've decoupled OCaml projects in Nix from the nixpkgs repository. | 15:49:40 |
| @timdeh:matrix.org | So the way most of the 2nix wrappers work is essentially what you describe yes. A standalone tool that can be used to convert some sort of lock or package format to nix fetch expressions. Then, typically, somebody integrates that tool into nixpkgs to create a repo of all (or most) of the available packages | 15:51:31 |
| @timdeh:matrix.org | There is also an open RFC which could help to improve this situation a bit, and make this type of thing more Nix native | 15:52:44 |
| d4hines | Then, typically, somebody integrates that tool into nixpkgs to create a repo of all (or most) of the available packages | 15:53:02 |
| d4hines | * Then, typically, somebody integrates that tool into nixpkgs to create a repo of all (or most) of the available packages Wait, how does that work? Let's say it's npm2nix. There are dozens of versions of package X on npm that live side-by-side, but nixpkgs has exactly 1 version per version of nixpkgs, right? | 15:53:15 |
| d4hines | *
Wait, how does that work? Let's say it's npm2nix. There are dozens of versions of package X on npm that live side-by-side, but nixpkgs has exactly 1 version per version of nixpkgs, right? | 15:53:24 |
| @timdeh:matrix.org | Yes, although some times there are multiple versions of the repo for different versions of node. I think atm there is nodePackages and nodePackages_latest. If I understand ericson's RFC correctly though, this would become a bit more dynamic and one could pull whichever version they like. | 15:56:00 |
| @timdeh:matrix.org | Since your colleague is already started with an overlay approach, you might at least be able to use nvfetcher as a stop gap until a more permanent solution arrives via either a proper 2nix wrapper, or the RFC mentioned materializes. It would at least make package source updates a little less painful. | 15:59:34 |
| d4hines | So let's talk about how nvfetcher would look:
| 16:05:24 |
| d4hines | As for our other projects, do you know off the top of your head if it's possible to build esy in Nix, and just use esy's functionality from there? I know there are some issues with this, namely that Esy also wants to control all the dependencies | 16:06:33 |
| @timdeh:matrix.org | Yes, you don't necessarily have to break out the overlay into a separate flake though. That's just a stylistic decision. At work we typically just stick to a single As for using esy as is with Nix, you could attempt to use a fixed-output derivation to give esy network access and generate whatever it generates. Don't know enough about it yet to know if that would work or be simple enough though. | 16:13:42 |
| d4hines | In general, are there strategies for an "escape hatch" with NIx? That's something that worries me before I fully switch over to NixOS. | 16:17:10 |
| d4hines |
Are there any open source examples you can share with me? I'd love to see an orchestration of nvfetcher, overlays, and flakes together. | 16:18:53 |
| @timdeh:matrix.org | The fixed-output derivation could be considered a type of escape hatch, where the derivation is allowed to make network calls under the condition that the final result matches a user supplied hash. There is also the buildFHS helper to simulate a traditional FHS environment when needed. It's also pretty simple to declaratively spin up oci or docker containers when you need them as well. | 16:21:38 |
| @timdeh:matrix.org | as for a good example of nvfetcher, I just recently integrated my devos install with it (it was merged fairly recently). I haven't even pushed it yet, because I was working out some errors yesterday, but it's fully working now so I can probably push it here in a bit and send you a link. I've also been thinking about integrating nvfetcher at work for some of our vendored packages, but I haven't started that yet. | 16:22:45 |
 David Arnold (blaggacao) | In reply to @d4hines:matrix.orgI can recommend https://www.tweag.io/blog/2021-03-04-gomod2nix/ | 16:23:03 |
| David Arnold (blaggacao) | Very interesting discussion! So an intermediate resume would be (in order of "nixonicness"): *2nix -> nvfetcher -> fixed output with target ecosystem tooling | 16:28:50 |
| David Arnold (blaggacao) | My takeaway here is that nvfetcher can actually sit somewhere in the middleground, here | 16:29:46 |
| David Arnold (blaggacao) | (if that makes sense) | 16:29:57 |
| d4hines | At this stage very little Nix usage makes sense to me 😆 | 16:30:31 |
| d4hines | I've read through some of the Nix pills and I've gotten DevOS working, and that's about it. It's quite a learning curve | 16:30:56 |
| d4hines | So any concrete examples would be super appreciated | 16:31:20 |
| David Arnold (blaggacao) | RFC92 fit's in in that it "defers" evaluation of an "input from derivation" that wraps ecosystem tooling (like esy, etc.) so long as as a text hash of those defered insturctions can be always expected to produce the same outpu. | 16:32:31 |
| David Arnold (blaggacao) | * RFC92 fit's in in that it "defers" evaluation of an "input from derivation" that wraps ecosystem tooling (like esy, etc.) so long as as a text hash of those defered ecosystem instructions can be always expected to produce the same output | 16:32:52 |