OfBorg - Public Room Timeline

	OfBorg	182 Members
	Number of builds and evals in queue: https://nix.ci/prometheus/graph?g0.expr=ofborg_queue_evaluator_waiting&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=2h&g1.expr=ofborg_queue_builder_waiting%7Barch!~%22.*-lowprior%22%7D&g1.tab=1&g1.stacked=0&g1.show_exemplars=0&g1.range_input=1w	63 Servers

Load older messages

Sender	Message	Time
3 Nov 2021
cole-h	I'm currently redeploying all the evaluators to run on ZFS, which should hopefully prevent the recent space issues (inode related) from reappearing. If you see any problems, please ping me directly either here or in any related issues / PRs.	17:41:06
	piegames set a profile picture.	22:01:14
cole-h	Just following up: it is done! All the evaluators are running off of ZFS now! Here's a little more backstory on the issue and why we decided to move the machines to ZFS: Originally, the machines were run off of EXT4. For years, it seems like this has been fine, but recently, we have been running into issues where the machines would complain about a lack of free disk space. When we went to go check, however, it wasn't disk space that was the problem, but available inodes! EXT4 has a limited amount of inodes (I believe it's somewhere around 4 billion?), and while derivations (e.g. the `.drv` files themeslves) are small, they each take up an inode (at least). Although the garbage collector does know how to "free X amount of data", it doesn't know how to make sure it frees "X amount of inodes". This lead to the disk having plenty of space, but absolutely no inodes available. Enter ZFS: ZFS gives you unlimited inodes (in that it doesn't have a fixed number of them available) so long as you have the disk space to support it. Then, in order to actually get the machines running ZFS, I had to do a few things: Use Equinix Metal's new NixOS support to deploy the instances Set up `customdata` in order to set up the ZFS pool using the disks exposed to the instance Deploy the instances one-by-one in order to verify they worked properly and wouldn't fall over if left alone For the moment, things seem to be working just fine! Fingers crossed, no more running-out-of-space alerts until we're actually running out of space... That said, I will still keep my eye on it. Once again, if you notice anything out of the ordinary, don't be a stranger: ping me (here or on the related issue / PR)!	22:10:27
cole-h	* Just following up: it is done! All the evaluators are running off of ZFS now! Here's a little more backstory on the issue and why we decided to move the machines to ZFS: Originally, the machines were run off of EXT4. For years, it seems like this has been fine, but recently, we have been running into issues where the machines would complain about a lack of free disk space. When we went to go check, however, it wasn't disk space that was the problem, but available inodes! EXT4 has a limited amount of inodes (I believe it's somewhere around 4 billion?), and while derivations (e.g. the `.drv` files themeslves) are small, they each take up an inode (at least). Although the garbage collector does know how to "free X amount of data", it doesn't know how to make sure it frees "X amount of inodes". This lead to the disk having plenty of space, but absolutely no inodes available. Enter ZFS: ZFS gives you unlimited inodes (in that it doesn't have a fixed number of them available) so long as you have the disk space to support it. Then, in order to actually get the machines running ZFS, I had to do a few things: Use Equinix Metal's new NixOS support to deploy the instances Set up `customdata` in order to set up the ZFS pool using the disks exposed to the instance Deploy the instances one-by-one in order to verify they worked properly and wouldn't fall over if left alone For the moment, things seem to be working just fine! Fingers crossed, no more running-out-of-space alerts until we're actually running out of space... That said, I will still keep my eye on it. Once again, if you notice anything out of the ordinary, don't be a stranger: ping me (here or on the related issue / PR)!	22:10:31
cole-h	* Just following up: it is done! All the evaluators are running off of ZFS now! Here's a little more backstory on the issue and why we decided to move the machines to ZFS: Originally, the machines were run off of EXT4. For years, it seems like this has been fine, but recently, we have been running into issues where the machines would complain about a lack of free disk space. When we went to go check, however, it wasn't disk space that was the problem, but available inodes! EXT4 has a limited amount of inodes (I believe it's somewhere around 4 billion?), and while derivations (e.g. the `.drv` files themeslves) are small, they each take up an inode (at least). Although the garbage collector does know how to "free X amount of data", it doesn't know how to make sure it frees "X amount of inodes". This lead to the disk having plenty of space, but absolutely no inodes available. Enter ZFS: ZFS gives you unlimited inodes (in that it doesn't have a fixed number of them available) so long as you have the disk space to support it. Then, in order to actually get the machines running ZFS, I had to do a few things: Use Equinix Metal's new NixOS support to deploy the instances Set up `customdata` in order to set up the ZFS pool using the disks exposed to the instance Deploy the instances one-by-one in order to verify they worked properly and wouldn't fall over if left alone For the moment, things seem to be working just fine! Fingers crossed, no more running-out-of-space alerts until we're actually running out of space... That said, I will still keep my eye on it. Once again, if you notice anything out of the ordinary, don't be a stranger: ping me (here or on the related issue / PR)!	22:10:40
cole-h	...that's a long message	22:11:00
	r-burns joined the room.	22:18:25
	jbedo joined the room.	22:50:40
	ryantm joined the room.	23:08:55
	Church joined the room.	23:14:00
	Janne Heß joined the room.	23:32:38
	Janne Heß left the room.	23:32:57
4 Nov 2021
	illustris joined the room.	02:36:26
	illustris left the room.	02:37:18
5 Nov 2021
r-burns	It looks like ofborg-eval-lib-tests fails on some PRs which modify stdenv setup.sh, is this a known issue? https://github.com/NixOS/nixpkgs/pull/144749#issuecomment-962029150	20:22:36
r-burns	* It looks like ofborg-eval-lib-tests fails on some PRs which modify stdenv `setup.sh`, is this a known issue? https://github.com/NixOS/nixpkgs/pull/144749#issuecomment-962029150	20:23:04
6 Nov 2021
cole-h	Hm, that kinda looks like https://github.com/NixOS/nixpkgs/issues/64459. And it makes sense, seeing as we recently switched the evaluators / build machines to ZFS......	00:08:00
cole-h	Rebuilding coreutils on my ZFS system to see if it appears there as well.	00:10:41
r-burns	Hmm I'd be surprised if that's the cause, my zfs box does stdenv rebuilds for breakfast	00:11:30
cole-h	Could be a configuration thing.	00:11:40
cole-h	Yeah, rebuilt on my system and nada.	00:15:48
cole-h	(nada being no failures)	00:16:03
cole-h	Attempting to build on that machine myself to see if it is, somehow, a transient failure.	00:24:54
cole-h	...	00:28:02
cole-h	Well, coreutils built fine. Now to build all of lib-tests.	00:32:53
cole-h	Hm, maybe vcunat is right: https://github.com/NixOS/nixpkgs/pull/142602#issuecomment-961787154	00:37:48
r-burns	But this wasn't happening before, right? Or were we just not aware of it	00:56:14
cole-h	It's the first I'm hearing of it, at least.	00:59:22
cole-h	And indeed, it succeeded building on that ofborg machine...	01:06:05
cole-h	btw, thanks for prompting me to look at that, Ryan Burns -- it's on my radar at the very least. Hopefully it doesn't appear too often (if ever again...)	01:21:42

Show newer messages

Back to Room ListRoom Version: 6