I've seen that occur a few times in other projects on buildbot as well.

I'm wondering if we can group our tests a little more whether that'd help; that way there'd be a smaller number of bigger builds.

We split them up so much originally to reduce RAM usage. But it's ended up causing other issues, such as many small builds waiting for builders to become available.

For now the rebuilds aren't too much strain on our resources.

That's reassuring!

e.g. smarter scheduling and making no op rebuilds faster.

Yeah only rebuilding the failed builds would be good. Cancelling an eval & its builds when a new one is triggered on the same PR could be worthwhile too, at least on repos where the builds are heavy?