In reply to @woobilicious:matrix.org
I guess linux-testing has some packaging changes. complaining about missing libtraceevent

looking for some feedback on what we should do with linuxPackages.ena (the driver for the ec2 nitro NICs). It's a bit complicated:

1. ena has an upstreamed driver, available for a very long time and which gets regular updates in mainline, that will work for the large majority of cases. https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amazon/ena

2. Development for that out-of-tree driver is released independently on a separate release cadence from the kernel here: https://github.com/amzn/amzn-drivers/ . It has more-frequent updates than what you can get on a stable kernel, and also contains some patches were rejected by upstream but are associated with performance improvements.

3. I think it's also worth mentioning that there's a sort-of third variant of the driver in the form of https://github.com/amazonlinux/linux where they aggressively and quickly pick commits from (2) into the in-tree module.

Problems with this:

1. Up until a few months ago, I was on the fw team for ena express and was paying close attention to all the driver releases for testing reasons, so I made an effort to make a PR to bump the ena module whenever I knew there was a new driver out, but A: it can be hard to get reviews and so they would tend to sit for a while, especially for a package like this that causes so much rebuilding, and 2. I now work more closely to EFA than ENA and I'm not able to stay on top of it anymore, so we can fall behind pretty quickly (as we have right now, see https://github.com/NixOS/nixpkgs/pull/313941 )

2. when nixpkgs does fall behind from the latest release, if someone is using a newer mainline-ish kernel, there's a chance that we're effectively downgrading the module for them and the in-tree module has bugfixes that they're missing.

3. I believe there's also the potential that the preference is not correct and the in-tree version gets loaded when it should not be, I need to check what happens here.

I think my preference is:

1. continue to have linuxPackages.ena, but make it opt-in. Should setup a nixpkgs-update script for it to watch for releases so we're sure it stays up to date, and document that it's only there if you're on a non-default kernel with a very outdated in-tree ena module. Ideally, restrict hydra to not build it everywhere, making the PRs more palatable to be merged quickly. People can opt into it if they prefer to, but just remove it from the default EC2 profile.

2. the ec2 profile in nixpkgs should prefer to use the amazonlinux kernel from (3), as it contains fixes for a lot of other ec2-specific things, and it also brings in newer ena drivers, too.

looking for some feedback on what we should do with linuxPackages.ena (the driver for the ec2 nitro NICs). It's a bit complicated:

1. ena has an upstreamed driver, available for a very long time and which gets regular updates in mainline, that will work for the large majority of cases. https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amazon/ena
2. Development for that out-of-tree driver is released independently on a separate release cadence from the kernel here: https://github.com/amzn/amzn-drivers/ . It has more-frequent updates than what you can get on a stable kernel, and also contains some patches were rejected by upstream but are associated with performance improvements.
3. I think it's also worth mentioning that there's a sort-of third variant of the driver in the form of https://github.com/amazonlinux/linux where they aggressively and quickly pick commits from (2) into the in-tree module.

Problems with this:

1. Up until a few months ago, I was on the fw team for ena express and was paying close attention to all the driver releases for testing reasons, so I made an effort to make a PR to bump the ena module whenever I knew there was a new driver out, but A: it can be hard to get reviews and so they would tend to sit for a while, especially for a package like this that causes so much rebuilding, and 2. I now work more closely to EFA than ENA and I'm not able to stay on top of it anymore, so we can fall behind pretty quickly (as we have right now, see https://github.com/NixOS/nixpkgs/pull/313941 )
2. when nixpkgs does fall behind from the latest release, if someone is using a newer mainline-ish kernel, there's a chance that we're effectively downgrading the driver for them and the in-tree module has bugfixes that they're missing in the out-of-date out-of-tree module.
3. I believe there's also the potential that the preference is not correct and the in-tree version gets loaded when it should not be (might not be present in the initrd by default on all the profiles...), I need to check what happens here.

looking for some feedback on what we should do with linuxPackages.ena (the driver for the ec2 nitro NICs). It's a bit complicated:

1. ena has an upstreamed driver, available for a very long time and which gets regular updates in mainline, that will work for the large majority of cases. https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amazon/ena
2. Development for that driver is released independently on a separate release cadence from the kernel here as an out-of-tree module: https://github.com/amzn/amzn-drivers/ . This is what linuxPackages.ena represents. It has more-frequent updates than what you can get on a stable kernel, and also contains some patches were rejected by upstream but are associated with performance improvements.
3. I think it's also worth mentioning that there's a sort-of third variant of the driver in the form of https://github.com/amazonlinux/linux where they aggressively and quickly pick commits from (2) into the in-tree module.

Problems with this:

1. Up until a few months ago, I was on the fw team for ena express and was paying close attention to all the driver releases for testing reasons, so I made an effort to make a PR to bump the ena module whenever I knew there was a new driver out, but A: it can be hard to get reviews and so they would tend to sit for a while, especially for a package like this that causes so much rebuilding, and 2. I now work more closely to EFA than ENA and I'm not able to stay on top of it anymore, so we can fall behind pretty quickly (as we have right now, see https://github.com/NixOS/nixpkgs/pull/313941 )
2. when nixpkgs does fall behind from the latest release, if someone is using a newer mainline-ish kernel, there's a chance that we're effectively downgrading the driver for them and the in-tree module has bugfixes that they're missing in the out-of-date out-of-tree module.
3. I believe there's also the potential that the preference is not correct and the in-tree version gets loaded when it should not be (might not be present in the initrd by default on all the profiles...), I need to check what happens here.

looking for some feedback on what we should do with linuxPackages.ena (the driver for the ec2 nitro NICs). It's a bit complicated:

1. ena has an upstreamed driver, available for a very long time and which gets regular updates in mainline, that will work for the large majority of cases. https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amazon/ena
2. Development for that driver is released independently on a separate release cadence from the kernel here as an out-of-tree module: https://github.com/amzn/amzn-drivers/ . This is what linuxPackages.ena represents. It has more-frequent updates than what you can get on a stable kernel, and also contains some patches were rejected by upstream but are associated with performance improvements.
3. I think it's also worth mentioning that there's a sort-of third variant of the driver in the form of https://github.com/amazonlinux/linux where they aggressively and quickly pick commits from (2) into the in-tree module.

Problems with this:

1. Up until a few months ago, I was on the fw team for ena express and was paying close attention to all the driver releases for testing reasons, so I made an effort to make a PR to bump the ena module whenever I knew there was a new driver out, but A: it can be hard to get reviews and so they would tend to sit for a while, especially for a package like this that causes so much rebuilding, and B: I now work more closely to EFA than ENA and I'm not able to stay on top of it anymore, so we can fall behind pretty quickly (as we have right now, see https://github.com/NixOS/nixpkgs/pull/313941 )
2. when nixpkgs does fall behind from the latest release, if someone is using a newer mainline-ish kernel, there's a chance that we're effectively downgrading the driver for them and the in-tree module has bugfixes that they're missing in the out-of-date out-of-tree module.
3. I believe there's also the potential that the preference is not correct and the in-tree version gets loaded when it should not be (might not be present in the initrd by default on all the profiles...), I need to check what happens here.