| 19 Mar 2022 |
 hexa | https://github.com/NixOS/nixpkgs/pull/164601 unwilling to eval apparently | 14:08:37 |
| hexa |
fatal: couldn't find remote ref pull/164601/head
o_O
| 14:09:02 |
 cole-h | Oh boy... | 17:45:39 |
| cole-h | Ah, found the real issue:
This SHA and context has reached the maximum number of statuses. | 17:50:38 |
| hexa | what does that even mean | 17:52:12 |
| cole-h | It likely means, when GitHub was having their bad day, that the API was doing weird things and caused ofborg to continuously attempt to add statuses or something | 17:53:14 |
| cole-h | https://api.github.com/repos/NixOS/nixpkgs/statuses/4e07eedf4dc9dbba87d3591271d158ceb9341f03?per_page=100 | 17:56:15 |
| cole-h | : ~ ; curl -v "https://api.github.com/repos/NixOS/nixpkgs/statuses/4e07eedf4dc9dbba87d3591271d158ceb9341f03?per_page=1" 2>&1 | rg -i 'link:'
< link: <https://api.github.com/repositories/4542716/statuses/4e07eedf4dc9dbba87d3591271d158ceb9341f03?per_page=1&page=2>; rel="next", <https://api.github.com/repositories/4542716/statuses/4e07eedf4dc9dbba87d3591271d158ceb9341f03?per_page=1&page=1167>; rel="last"
| 17:57:23 |
| cole-h | So that commit has ~1167 statuses on it lol | 17:57:36 |
| cole-h | This same thing happened in the past, and I sent in a support ticket and got the above hint to look for how it's exceeding the maximum number of statuses (1000) | 17:59:31 |
| cole-h | hexa: Fetching from the pull head ref should work now :) | 18:00:57 |
| hexa | ❤️ | 18:03:45 |
| 23 Mar 2022 |
 Sandro | In reply to @7c6f434c:nitro.chat
Graham has said many times (and correctly), that even a relatively low rate of false-positive-red will devalue red for very clear failures. Build test failures… vary in their expected impact on decisions and how often to we muss grey checks by accident and break things? | 10:56:12 |
| Sandro | Either I stressed ofborg to much with staging PRs or GitHub is having issues again. Checks are taking very long... | 10:57:47 |
 7c6f434c | In reply to @sandro:supersandro.de
and how often to we muss grey checks by accident and break things? It's GitHub, you cannot have things work actually as they should with GitHub | 11:06:29 |
 piegames | In reply to @7c6f434c:nitro.chat
It's GitHub, you cannot have things work actually as they should with GitHub cries in CODEOWNERS… | 11:07:20 |
| Sandro | In reply to @7c6f434c:nitro.chat
It's GitHub, you cannot have things work actually as they should with GitHub Its not as terrible as everyone is saying. Launchpad or gerrit have their flaws, too. | 11:26:26 |
| hexa | https://www.githubstatus.com/incidents/tyc8wpsgr2r8 | 15:07:27 |
| hexa | posting here just in case people are wondering what's up | 15:07:33 |
| cole-h | I should probably start drafting a comment to mass post once it's fixed... lol | 15:09:28 |
| 24 Mar 2022 |
| hexa | so now if they want to shed load they'll do it on the webhooks front | 15:14:28 |
| hexa | that's … not good. | 15:14:30 |
| hexa | taken from https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ | 15:22:02 |
| hexa |
In order to reduce load, we throttled webhook traffic and will continue to use that as a mitigation to prevent future recurrence during peak load times as we continue to investigate further mitigations.
| 15:22:17 |
| cole-h | Guh | 15:23:15 |
| cole-h | Going to be redeploying ofborg to pick up https://github.com/NixOS/ofborg/pull/588 and give that a shot. Cross your fingers. | 20:39:07 |
| cole-h | As always, let me know if you see any weirdness :) | 20:39:33 |
| 26 Mar 2022 |
| Sandro | In reply to @hexa:lossy.network
taken from https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ It seems to be that a lot of downtimes are due to their database | 09:23:44 |
| 27 Mar 2022 |
|  @grahamc:nixos.org invited  Artturin. | 23:19:12 |
| Artturin joined the room. | 23:19:17 |
