| 27 Jun 2025 |
 vcunat | Or maybe the build really does get stuck and that's why it times out after 2h. | 21:06:03 |
| 28 Jun 2025 |
 hexa | firefox 140.0.1 | 00:51:06 |
| hexa | built for 24.11 | 00:51:11 |
| hexa | takes just under 2h49m to build and complete the nixos test | 00:51:30 |
| hexa | years ago when I started maintaining it I could build firefox in under an hour easily | 00:51:56 |
| hexa | like 50m on my desktop | 00:52:07 |
| hexa | pgo made that worse, but its was worth it for the performance boost | 00:52:23 |
| hexa | so 2x 50m + 15m profiling, just shy of two hours | 00:52:56 |
| hexa | but now we're closer to three hours 🫠 | 00:53:04 |
| hexa | so indeed 4h49m sounds like 2h49 + 2h silent timeout 😬 | 13:11:10 |
| hexa | wondering if bumping that up from 2 to 3 hours would make it work | 13:11:27 |
| hexa | updated the linux builders | 13:18:12 |
| vcunat | Indeed. On the last restart I looked at the tail of the log periodically, and it really looks like the 2h silent timeout. Always in the same place. | 15:59:19 |
| vcunat | Ah, that was step 7. And the step 8 also looks stuck the same now. | 16:04:33 |
| vcunat | At this point I don't think more retries will help. | 16:04:56 |
| hexa | Receiving outputs | 16:12:41 |
| hexa | 2h48m | 16:12:48 |
| vcunat | 🎉🎉🎉 | 16:21:36 |
| vcunat | Though there's some more general problem around this on 24.11. I see the same with other variants: devedition, beta, thunderbird-latest, librewolf. | 16:35:50 |
 emily | I feel the silent timeout is too short in general | 16:38:13 |
| emily | but maybe you could turn on some spammy logging in the build like we did with LLVM | 16:38:22 |
 aloisw | That won't help when the linker is swapping for over 1 hour because it uses 50 gigabytes of RAM. | 16:38:52 |
| emily | make it print every time it dereferences a pointer :P | 16:40:04 |
| emily | do we link browsers with the GNU linker currently? | 16:40:20 |
| emily | if yes, we should consider… not doing that | 16:40:25 |
| aloisw | All the Geckos I have seen building linked using lld. | 16:40:54 |
| aloisw | It is unclear whether they are supposed to use 50 gigabytes, as https://firefox-source-docs.mozilla.org/setup/linux_build.html states 4 GB should be enough although 8 are recommended. | 16:43:01 |
| emily | I disbelieve 4 GiB. how long ago was that written | 16:44:06 |
| aloisw | I don't doubt it if you have enough swap and time. Although the 30 GB of disk are not enough in this case because even the swap needed is more. | 16:46:39 |
| vcunat | Swap usage is minimal. (all these builds run just on this particular machine)
https://grafana.nixos.org/d/rYdddlPWk/node-exporter-full?orgId=1&var-DS_PROMETHEUS=default&var-job=node&var-node=hopeful-rivest.builder.nixos.org:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&from=now-24h&to=now&var-datasource=default&refresh=1m&timezone=browser&viewPanel=panel-78 | 16:48:31 |
