| 22 Jun 2021 |
 David Arnold | https://demo.hedgedoc.org/s/RO9YawHcY# | 20:03:32 |
| David Arnold | (ideas worth spreading?) | 20:03:51 |
 tomberek | In reply to @blaggacao:matrix.org
https://demo.hedgedoc.org/s/RO9YawHcY# I enjoyed this article. https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/ I prefer to take it farther and even consider some batch data flows to be “indistinguishable from a build/CI system”. And it just so happens we have a powerful+flexible build system to utilize. | 20:13:50 |
| David Arnold | Thanks I'll put that in the intro! | 20:18:45 |
| David Arnold | Nice! That guy had the same feeling... I'd like for that article to be updated and instead of directing towards taskcluster, let them shiver in awe for what can be done with nix and a declarative State Machine + a declarative rules evaluator. | 20:23:48 |
| David Arnold | * Nice! That guy had the same feeling... I'd like for that article to be updated and instead of directing towards taskcluster, let them shiver in awe for what can be done with nix (the build DAG) and a declarative State Machine + a declarative rules evaluator. | 20:26:27 |
| David Arnold | AM I correct that we are slowly working towards fanning out builds within a DAG in nix? | 20:28:33 |
| David Arnold | * Am I correct that we are slowly working towards fanning out builds to remote builders within a DAG in nix? | 20:28:51 |
| David Arnold | * Am I correct that we are slowly working towards fanning out builds to remote builders within a build DAG in nix? | 20:28:57 |
| David Arnold | On the partial build side of things, where are we standing vs. bazel? Do we have already have composable partial rebuilds according to subtree changes? | 20:30:25 |
| David Arnold | I like the hydraJobs flake output special meaning, but I'd prefer it to be just jobs with a well informed interface. | 20:32:30 |
| David Arnold | * I like the hydraJobs flake output special meaning, but I'd prefer it to be just jobs with a well informed interface, that not only can return derivations but also talke to a remote resource (the state machine). Ideas? | 20:33:12 |
| David Arnold | Looks like there is potential for entanglement ... | 20:33:46 |
| David Arnold | * I like the hydraJobs flake output special meaning, but I'd prefer it to be just jobs with a well informed interface, that not only can return derivations but also talk to a remote resource (the state machine). Ideas? | 20:34:06 |
| tomberek | The granularity (or lack of) of some builds is the primary limitation for partial rebuilds. I’ve thought of a check-pointing mechanism for certain builds would be helpful, but I don’t know how to make it general enough. I’m also under the impression some PoCs have been done with bazel and a few others. | 20:38:25 |
| David Arnold | Thanks! I think flakes with their very strict input / output interfaces can be a milestone towards granularity. What do you think? | 20:42:58 |
| David Arnold | Also I'm breading over this addition:
## `taskcluster`? (pha, `nix` build DAG!)
Let's circle back to `taskcluster` as mentioned in the introductory rant. It seems that task cluster knows how to run a DAG of batch jobs. That's a bit of a different focus than a proper state machine, though. As for the build itself, we have our beloved `nix` primitives to resolve the build DAG.
| 20:43:32 |
| David Arnold | I'm not familiar enough with how everything is currently working, but would it make sense to wish for
some remote build delegation and parallelization while observing the inherent build DAG.
| 20:44:43 |
| David Arnold | (like builtin into nix itself, if that's not already the case?) | 20:44:59 |
 Rick (Mindavi) | Content-addressability will ensure that partial rebuilds will work (when it's determined that there's no influence on the build) | 20:47:27 |
| Rick (Mindavi) | Nix will indeed build in parallel when it's possible | 20:47:43 |
| Rick (Mindavi) | Taking the graph into account | 20:47:56 |
| David Arnold | Cool! | 20:48:05 |
| David Arnold |
nix already knows how to parrallelize when possible.
| 20:48:26 |
| David Arnold | Feel free to add / or reword to make it more concise: https://demo.hedgedoc.org/sfTFtgceQJqI2gPqIwXQ5Q#taskcluster-pha-nix-build-DAG | 20:49:48 |
| David Arnold | The point to bring about is: nix is task cluster in the build domain, and everything else is better handled by a proper state machine, not a simple DAG. | 20:50:29 |
| tomberek | I’m not understanding where you intend state would be kept. Or where this state machine would run. (I’ve found it possible, I’m just not grokking specifically what you intend.) | 21:41:19 |
| David Arnold | I am a bit bisased, since I have somewhat knowledge of how zeebe works: it's a state machine that otherwise does event sourcing into a data lake. +- | 22:04:10 |
| David Arnold | It's messages all the way down, and they modify constitute the state. | 22:05:07 |
| tomberek | I pulled a trick recently where I put state into Git (it was very slow moving state, human time-scales) and Hydra would pick it up and run a DAG of the changed transformations. Along with an auto-scaling group for builders, it made a data ingestion system with many nice qualities inherited from Nix. | 22:24:45 |
