!djTaTBQyWEPRQxrPTb:nixos.org

Nixpkgs Architecture Team

228 Members
https://github.com/nixpkgs-architecture, weekly public meetings on Wednesday 15:00-16:00 UTC at https://meet.jit.si/nixpkgs-architecture53 Servers

Load older messages


SenderMessageTime
12 Jul 2022
@yorik.sar:matrix.orgyorik.sarIf we go this way, we could consider putting stdenvs and their dependencies into separate "sets".11:45:35
@kevincox:matrix.orgkevincox
In reply to @k900:0upti.me
Maybe in a special top-level directory
Or just python3Packages is a "regular" package in terms of the first lookup. So maybe it is py/python3Packages.nix but then it can do it's own lookup, maybe using the same scheme or maybe using a different one.
11:55:46
13 Jul 2022
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusRedacted or Malformed Event10:31:53
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius

I noticed that I sparked quite a discussion 😄 So I ideally would see:

  • Folder structure representing attrs structure. If package is in top level - under root directory. If there is a package group (gnome, pythonPackages, etc) - then it should be under directory with the same name
  • I got a feeling that all-packages.nix is necessary otherwise we would be impacting performance but I have not tested it (also we could autogenerate such file)
  • all these overrides should be separated from all-packages.nix I'd say one file for 'dumb package definitions' and another for overwrites
  • Consider exposing blueprints https://github.com/gytis-ivaskevicius/nix-patterns/blob/master/blueprints/flake.nix#L7-L10 instead of derivations and mapAttrs when necessary (is it more work than its worth?)0
10:35:37
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius
nix-repl> length (attrNames pkgs)
16563
10:36:29
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius👀10:36:36
@kevincox:matrix.orgkevincox I feel that having an all-packages.nix can't exist for performance reasons because we don't want to parse and evaluate this ever growing file. If you can just resolve an attr into a file and load it you avoid having to load that into memory and the cost is only the definitions of the closure. 10:38:09
@kevincox:matrix.orgkevincox

I wonder if we need overrides in nixpkgs. Maybe we can give every modified package a name and let the called package use that name. (basically no more python = python38). But that does add some API instability for user overrides. Maybe a multi-level approach can work?

I like the look of blueprints but how does this work for user-overrides?

10:40:56
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusI think we should first provide options to not to use overwrites and maybe few years down the line we can remove those alltogether10:41:43
@k900:0upti.meK900 I feel like we need something like overrideAttrs, but stuff like override should be exposed as (args) -> derivation functions 10:41:54
@kevincox:matrix.orgkevincox I think we need to provide user overrides. For example I use a custom libvte in my system and it is important to me that all packages pick that up. 10:42:56
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius
In reply to @k900:0upti.me
I feel like we need something like overrideAttrs, but stuff like override should be exposed as (args) -> derivation functions
I would like to try to push concept as 'blueprints' (link above) as a native nix structure. First we need to implement it on multiple flakes and see how community reacts
10:43:32
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusand its basically what you are looking for10:43:51
@gytis-ivaskevicius:matrix.orgGytis Ivaskevicius how would one go about benchmarking all-packages.nix so first step would be to remove all lines with callPackage, then setup some autodetection logic, but then what? 10:46:32
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciuswhats the proper benchmark scenario?10:46:44
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusbuilding one package?10:47:01
@kevincox:matrix.orgkevincox

I think there are probably two main scenarios.

  1. Building a small closure with only a few deps.
  2. Building a large closure (a full-featured desktop NixOS system with lots of packages installed?)

I think having all-packages.nix probably doesn't hurt 2 much but costs 1 a lot.

11:15:24
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusI dont see how second scenario differs. we should see the same performance change11:30:40
@kevincox:matrix.orgkevincox 1 Benefits from lower fixed-cost overhead while 2 benefits from lower per-lookup overhead. 11:32:36
@gytis-ivaskevicius:matrix.orgGytis IvaskeviciusI feel dumb suggesting this but how does it sound to have a folder for each letter of alphabeth and place projects depending on the letter they start with 😄12:21:51
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusthats a 'fix' for github 1000 files limit12:22:14
@infinisil:matrix.orginfinisil Gytis Ivaskevicius: This was discussed a bunch if you scroll up :) 12:22:33
@gytis-ivaskevicius:matrix.orgGytis Ivaskeviciusah great, at least im not the only one considering that 😄12:22:59
@infinisil:matrix.orginfinisil Another idea I played around with is to extend the notion of packages and their versions. One of the goals of it is to clean up handling of versions, aka getting rid of the python, python2, python27, ffmpeg, ffmpeg_full, etc. inconsistencies 12:33:51
@infinisil:matrix.orginfinisil Main idea is to allow each package name to be associated with arbitrarily many versions. The interface might look something like python (default), python.version "2", python.version "2.7", ffmpeg, ffmpeg.version "full" 12:35:24
@kevincox:matrix.orgkevincox
In reply to @infinisil:matrix.org
Another idea I played around with is to extend the notion of packages and their versions. One of the goals of it is to clean up handling of versions, aka getting rid of the python, python2, python27, ffmpeg, ffmpeg_full, etc. inconsistencies
Maybe I lack creativity but I think that would be a different project. For now they will just end up in python.nix, python2.nix... Those files could just be trivial import stubs.
12:35:34
@infinisil:matrix.orginfinisilWell it is another idea, but it plays into the above ideas, let me explain a bit more12:36:19
@infinisil:matrix.orginfinisilAs implied with the "full", this can go a bit further, versions don't need to be a single string, but it can be something more abstract. A version could be a combination of: Multiple forks of a project, multiple semantic versions, different variants and more12:38:01
@kevincox:matrix.orgkevincox
In reply to @kevincox:matrix.org
I think we need to provide user overrides. For example I use a custom libvte in my system and it is important to me that all packages pick that up.

Also for overrides I think we can probably handle this basically the same we do today in all-packages.nix for now. We can think about changing those as a different project. For now it could look something like:

# foo.nix
{pkgs}: pkgs.callPackage foo/impl.nix { python = pkgs.python27 }

That isn't the prettiest but allows us to do overrides and maintains the current API. Then I think we can probably clean it up as a separate effort if we wanted too. Unless I am also missing a way that merging both projects returns a better result here.

12:38:22
@kevincox:matrix.orgkevincox
In reply to @infinisil:matrix.org
As implied with the "full", this can go a bit further, versions don't need to be a single string, but it can be something more abstract. A version could be a combination of: Multiple forks of a project, multiple semantic versions, different variants and more
At this point how is this different from regular package arguments that can be overridden? Sounds like you can have python.override { version = "2.7" } if we anted to.
12:39:48

Show newer messages


Back to Room ListRoom Version: 9