!RbXGJhHMsnQcNIDFWN:nixos.org

Haskell in Nixpkgs/NixOS

734 Members
For discussions and questions about Haskell with Nix, cabal2nix and haskellPackages in nixpkgs | Current Docs: https://haskell4nix.readthedocs.io/145 Servers

Load older messages


SenderMessageTime
26 May 2021
@sternenseemann:systemli.orgsterni (he/him)
In reply to @maralorn:maralorn.de
From all I can tell peti never gave a frack about rebuilds. Just merge it.^^
ideally you would merge master into haskell-updates once again wait fro the rebuilds to finish and merge then
12:39:53
@sternenseemann:systemli.orgsterni (he/him)which has the same outcome really, but avoids blocking the channel by accident (I believe cachix is a channel constituent)12:40:14
@sternenseemann:systemli.orgsterni (he/him)
In reply to @maralorn:maralorn.de
The problem is that this would be a moderately invasive operation on cabal2nix/hackage2nix and none of us maintainers has a lot of free time or familarity with the code base. So any outside help on this topic would be appreciated.
also this is yet another layer of complexity added to the process we have currently which we already made quite a bit more involved than before (in terms of moving / interacting parts, not necessarily usability)
12:41:21
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
also this is yet another layer of complexity added to the process we have currently which we already made quite a bit more involved than before (in terms of moving / interacting parts, not necessarily usability)
I am not really afraid of that. Yes, automation brings complexity but if done well complexity covered by the code is complexity we developers don't need to deal with.
12:57:56
@sternenseemann:systemli.orgsterni (he/him)It's not just code complexity it's also semantic complexity you have to kinda be aware of when contributing12:59:23
@sternenseemann:systemli.orgsterni (he/him)at least that is what I am afraid of we'll reach12:59:32
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
which has the same outcome really, but avoids blocking the channel by accident (I believe cachix is a channel constituent)
If cachix builds on our branch and not on master then in 99% of cases that means that cachix is already broken on master. So in a sense that's not our fault. It should be fixed as soon as possible but that's orthogonal to our PR. Besides build fails channel advancement depends on all package build being finished so that does not have a lot to do with cachix.
13:02:32
@sternenseemann:systemli.orgsterni (he/him)
In reply to @maralorn:maralorn.de
If cachix builds on our branch and not on master then in 99% of cases that means that cachix is already broken on master. So in a sense that's not our fault. It should be fixed as soon as possible but that's orthogonal to our PR. Besides build fails channel advancement depends on all package build being finished so that does not have a lot to do with cachix.
not about failures, build times
13:03:02
@sternenseemann:systemli.orgsterni (he/him)like the channel will be unable to advance until we have rebuild enough of the haskell packages set to build cachix13:03:24
@sternenseemann:systemli.orgsterni (he/him)if it takes a day to rebuild we can just do it on the branch and have the cache fully populated already when we merge13:03:46
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
It's not just code complexity it's also semantic complexity you have to kinda be aware of when contributing
We actually already have the case that broken flags are kinda opaque to drive by contributers. Sometimes packages are marked broken without an easily understandable reason and cdepillabout pings me to figure out why. (this is more of a statement of fact. I am not arguing in any specific direction.)
13:05:26
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
like the channel will be unable to advance until we have rebuild enough of the haskell packages set to build cachix
I understand your point, but this has nothing to do with cachix. The channel will only advance when all jobs in the evaluation are finished. This includes all haskellPackages. Or are we talking about unstable-small and cachix is included in that? I assume it isn't.
13:07:18
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
if it takes a day to rebuild we can just do it on the branch and have the cache fully populated already when we merge
I am torn on this point. The fact is if our merge would mean a 500+ rebuild we would have to target staging. (and haskell builds aren't the fastest.) So we should basically adhere to that. Otoh the longer we push off a merge the higher is the risk that hydra does redundant additional work.
13:10:15
@sternenseemann:systemli.orgsterni (he/him)the channel will advance when all constituents of a certain aggregate job are built not when all of nixpkgs is built13:11:13
@sternenseemann:systemli.orgsterni (he/him)and cachix is part of the nixpkgs:unstable job13:11:23
@sternenseemann:systemli.orgsterni (he/him)https://hydra.nixos.org/job/nixpkgs/trunk/unstable13:11:59
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
and cachix is part of the nixpkgs:unstable job
Are you telling me that haskellPackages is not part of the nixos unstable job?
13:12:11
@maralorn:maralorn.demaralornWhat we actually need is ofBorg or some other tooling to tell us if the new derivations resulting from a merge are already cached in hydra. That should be totally doable by pulling the nar files.13:12:43
@sternenseemann:systemli.orgsterni (he/him)nixos:trunk-combined:tested tests a bunch of VM tests which are much more elaborate, so unstable is usually behind nixpkgs-unstable13:13:08
@sternenseemann:systemli.orgsterni (he/him)it probably depends on haskell in some way as well13:13:14
@sternenseemann:systemli.orgsterni (he/him) haskellPackages is part of the jobsets, but not directly of the aggregate job cnotrolling channel advancements 13:13:39
@sternenseemann:systemli.orgsterni (he/him)
In reply to @maralorn:maralorn.de
What we actually need is ofBorg or some other tooling to tell us if the new derivations resulting from a merge are already cached in hydra. That should be totally doable by pulling the nar files.
you could just check the drv outpaths before and after test merging — if they differ we have no cache
13:14:29
@maralorn:maralorn.demaralornYeah.true.13:14:56
@sternenseemann:systemli.orgsterni (he/him)I don't see why we shouldn't merge master into haskell-updates prior to a merge: It causes the same outcome as if we merged the branch into master, but we migitate the risks by doing the rebuilds without slowing the channel down and we even can catch unlikely regressions before the ruin someone's day13:15:57
@mjrosenb:matrix.orgMarty Rosenberghas anyone tried to use miso on a modern macos? the pinned nixpkgs don't seem happy.13:16:20
@sternenseemann:systemli.orgsterni (he/him) maralorn: cdepillabout: there has been a staging-next merge since the last time we merged master including changes to LLVM for example, so there will be rebuilds 13:17:08
@maralorn:maralorn.demaralorn
In reply to @sternenseemann:systemli.org
I don't see why we shouldn't merge master into haskell-updates prior to a merge: It causes the same outcome as if we merged the branch into master, but we migitate the risks by doing the rebuilds without slowing the channel down and we even can catch unlikely regressions before the ruin someone's day

Well my argument against it are 1. it will slow us down which is might be annoying for our process and for the person doing the merge who wants to get it done and 2. this could actually mean more unnecessary rebuilds. Imagine we have a change to a core package like random which leads to mass rebuilds and a small change to master which leads to us postponing the merge and then another change hits master leading to rebuild all packages again with the old "random" and then we need to rebuilt everything with the new random before we can merge. (I kinda dread the loop of merging in master, rebuilding, merging in master, rebuilding and never getting to actually merge. That’s theoretically possible.)

But those are only arguments why it isn‘t completely free to merge in master. When our merges causes a lot of rebuilds, yes, then we need to obey the same rules as the rest of nixpkgs and build them before merging into master.

And now that I think about it: Merging in master, retriggering an eval, seeing how many builds will be rebuild and merging when it’s less then 100 or something seems to be a matter of 10 minutes and is probably totally sane.

13:25:50
@domenkozar:matrix.orgDomen Kožar+1 for treating haskell-updates similar to staging-next, mass rebuilds so make sure there are big regressions and everything builds13:26:21
@domenkozar:matrix.orgDomen Kožarit does add overhead, but it's no worse than finding out the same stuff in master13:27:25
@domenkozar:matrix.orgDomen Kožarit's not so much overhead but rather delay13:28:45

Show newer messages


Back to Room ListRoom Version: 6