!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

280 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.91 Servers

Load older messages


SenderMessageTime
11 Oct 2024
@vcunat:matrix.orgVladimír ČunátRight. It might be risky in some ways.06:15:46
@emilazy:matrix.orgemilyI take it setting up a separate Hydra instance to test it would be onerous – we'd need to just switch over?06:17:32
@vcunat:matrix.orgVladimír ČunátI don't see a little down-time as an issue (for hydra.nixos.org)06:20:38
@emilazy:matrix.orgemilyI more meant it sounds like a pain to revert if it goes wrong06:22:16
@emilazy:matrix.orgemily(but maybe it's not)06:22:32
@vcunat:matrix.orgVladimír ČunátNo need to wipe the old machine too soon. And it seems unlikely that we'd want to revert completely.06:23:14
@tomberek:matrix.orgtomberek
In reply to @emilazy:matrix.org
if the Foundation will pay for a month then it seems worth it to just do it?
Not too hard. We're spending similar amounts on meetups/pizza. Infra team should have a discretionary budget (no?), otherwise throw a submission into this repo similar to this: https://github.com/NixOS/foundation/issues
06:39:11
@emilazy:matrix.orgemilyI didn't get the impression that the infra team has a budget at all06:39:33
@emilazy:matrix.orgemilythough I don't know how the finances work at all :)06:39:44
@emilazy:matrix.orgemily
In reply to @vcunat:matrix.org
No need to wipe the old machine too soon. And it seems unlikely that we'd want to revert completely.
I think we should do whatever you think is best for Hydra out of trying an existing AArch64 machine and getting an EPYC to try.
06:40:24
@emilazy:matrix.orgemily(without consideration of the costs, since they seem trivial compared to the potential benefit)06:40:42
@vcunat:matrix.orgVladimír ČunátKeeping it at x86 seems better in our current situation (less risks/complications).06:53:55
@vcunat:matrix.orgVladimír ČunátWe might consider splitting it up. At least the queue-runner away from the public web (hydra.nixos.org). I don't feel too comfortable having the signing keys on same machine as a public web. I think these only communicate through the DB which is on a separate machine already.06:55:56
@fricklerhandwerk:matrix.orgfricklerhandwerk
In reply to @joerg:thalheim.io
@fricklerhandwerk:matrix.org: where is this security tracker deployed just now? I think it would help if Erethon would have a chat with us, so we can make sure it get integrated with the rest of the infra to some extent.

http://sectracker.nixpkgs.lahfa.xyz/

Config here: https://github.com/Nix-Security-WG/nix-security-tracker/tree/main/staging

07:28:08
@glepage:matrix.orgGaétan LepageRedacted or Malformed Event07:30:30
@fricklerhandwerk:matrix.orgfricklerhandwerk
In reply to @emilazy:matrix.org
I didn't get the impression that the infra team has a budget at all
The infra team can get expenses reimbursed out of the foundation's regular budget. The problem is that in the past there was no process to do anything else. tomberek and I have been pushing to have discretionary budgets for teams, but there needs to be someone on the other end to make those decisions.
07:30:34
@emilazy:matrix.orgemilygotcha07:33:51
@emilazy:matrix.orgemily
In reply to @vcunat:matrix.org
We might consider splitting it up. At least the queue-runner away from the public web (hydra.nixos.org). I don't feel too comfortable having the signing keys on same machine as a public web. I think these only communicate through the DB which is on a separate machine already.
maybe the website could continue being on the x86 server?
07:34:24
@emilazy:matrix.orgemilyor, right, the idea was to upgrade the x8607:34:39
@emilazy:matrix.orgemilythen maybe the site could move to one of the AArch64 boxes? :)07:34:48
@emilazy:matrix.orgemilybtw, maybe should upgrade Hydra's Nix to 2.24 if it's moving to a new server anyway. since Hydra supports it now and it would solve that Nix bug that didn't get a fix backported because of only 2.18 and 2.24 being supported versions.07:51:44
@joerg:thalheim.ioMic92
In reply to @emilazy:matrix.org
btw, maybe should upgrade Hydra's Nix to 2.24 if it's moving to a new server anyway. since Hydra supports it now and it would solve that Nix bug that didn't get a fix backported because of only 2.18 and 2.24 being supported versions.
Yes. that was the plan. We recently fixed 2.24 support in hydra.
07:59:38
@joerg:thalheim.ioMic92
In reply to @fricklerhandwerk:matrix.org

http://sectracker.nixpkgs.lahfa.xyz/

Config here: https://github.com/Nix-Security-WG/nix-security-tracker/tree/main/staging

ok. Seems to be down just now.
07:59:56
@emilazy:matrix.orgemily(btw I never realized that the cache signing key is on the machine that hosts the hydra.nixos.org site)08:00:49
@emilazy:matrix.orgemily(that terrifies me)08:00:52
@emilazy:matrix.orgemily(strong support for any plan that fixes that)08:01:35
@fricklerhandwerk:matrix.orgfricklerhandwerk
In reply to @joerg:thalheim.io
ok. Seems to be down just now.
Yeah we're debugging it, there's some weird Django issues where some stupid script hangs itself to death
08:04:49
@dgrig:erethon.comdgrig
In reply to @joerg:thalheim.io
@fricklerhandwerk:matrix.org: where is this security tracker deployed just now? I think it would help if Erethon would have a chat with us, so we can make sure it get integrated with the rest of the infra to some extent.
I'm catching up with the channel now, for some reason Element didn't ping me about this mention (maybe because it was capitalized). Just want to point out that what fricklerhandwerk linked too is also Raito's work, don't want to take credit for it. At this point I'm not sure what's the best way to move forward, since it seems that Raito has already done a lot of the work needed to deploy this properly?
08:05:26
@fricklerhandwerk:matrix.orgfricklerhandwerk Mic92: back online 08:06:32
@joerg:thalheim.ioMic92What is your expected update cadence?08:06:41

Show newer messages


Back to Room ListRoom Version: 6