In reply to @katexochen:matrix.org
Grepping nix store for writable files returns a ton, both on my local system as well as on my remote builder:
find /nix/store -type f -perm -u+w ! -perm -g+w ! -perm -o+w
Those have also modified time stamps. 🤔

In reply to @katexochen:matrix.org

I can reproduce this using nixpkgs-review + remote builder with a commit that doesn't build. For example, running nixpkgs-review rev HEAD on https://github.com/katexochen/nixpkgs/tree/repro/azure-cli/2-65 will lead to the following on the configured remote builder:

$ ls -la /nix/store/*azure-cli* | grep nixbld
drwxrwxr-t 1 root nixbld 6669116 Oct 14 18:01 ..
drwxrwxr-t 1 root nixbld 6669116 Oct 14 18:01 ..
drwxrwxr-t 1 root      nixbld    6669116 Oct 14 18:01 ..
drwxrwxr-t 1 root nixbld  6669116 Oct 14 18:01 ..
drwxrwxr-t 1 root nixbld 6669116 Oct 14 18:01 ..

In reply to @raboof:matrix.org
interestingly this returns no results on my NixOS installation (which doesn't use store optimization). Are those files hardlinked by store optimization?

The one from above isn't:

$ find / -inum 189116731 2>/dev/null
/nix/store/a3b4xpykc7x1gz7d5kvcw40qf0xlqg4n-python3.11-fabric-3.2.2/lib/python3.11/site-packages/fabric-3.2.2.dist-info/top_level.txt

In reply to @atemu12:matrix.org
Does /nix/store/a3b4xpykc7x1gz7d5kvcw40qf0xlqg4n-python3.11-fabric-3.2.2.lock exist?

In reply to @katexochen:matrix.org
No explanation how the writable file ended up in the image, no. I'd have to look deeper into the hardlink optimization. Maybe there is a race with creating the link under /nix/store/.links and the build failing? Then any file from a successful derivation could be hardlinked against a writable file.

I searched for writable files that are hardlinked into a derivation that doesn't have a .lock, and it seems like there is none, so my theory doesn't work out:

#!/usr/bin/env bash

while read -r file; do
    links=$(stat -c %h "$file")
    if [[ "$links" -le 1 ]]; then
        echo "No links for $file"
        continue
    fi
    inode=$(stat -c %i "$file")
    while read -r linkedFile; do
        if [[ "$linkedFile" == "$file" ]]; then
            continue
        fi
        linkedDrv=$(echo "$linkedFile" | cut -d'/' -f1-4)
        if [[ -e "${linkedDrv}.lock" ]]; then
            echo "Lock file exists for $linkedDrv"
            continue
        fi
        echo "$linkedFile"
    done < <(find /nix/store -type f -inum "$inode" 2>/dev/null)
done < <(find /nix/store -type f -perm -u+w ! -perm -g+w ! -perm -o+w 2>/dev/null)

This is what we are trying to debug (from an uki/repart nixos image):

❯ diffoscope f*/initrd_0
--- f-btrfs/initrd_0
+++ f-ext/initrd_0
│┄ comprises of 1 embedded members
├── .cpio file embedded at offset 0
│ ├── file list
│ │ @@ -981,15 +981,15 @@
│ │  -r--r--r--   0        0        0     2274 1970-01-01 00:00:01.000000 ./nix/store/frngrlq2xa2szv4bx91jkymv0dy9akmx-kbd-2.6.4/share/keymaps/sun/sunt5-uk.map.gz
│ │  -r--r--r--   0        0        0     4926 1970-01-01 00:00:01.000000 ./nix/store/frngrlq2xa2szv4bx91jkymv0dy9akmx-kbd-2.6.4/share/keymaps/sun/sunt5-us-cz.map.gz
│ │  -r--r--r--   0        0        0     1706 1970-01-01 00:00:01.000000 ./nix/store/frngrlq2xa2szv4bx91jkymv0dy9akmx-kbd-2.6.4/share/keymaps/sun/sunt6-uk.map.gz
│ │  drwxr-xr-x   0        0        0        0 1970-01-01 00:00:01.000000 ./nix/store/fvsswlqh6pc2f5x9qvahm4qbjgjlp86j-linux-pam-1.6.1/
│ │  drwxr-xr-x   0        0        0        0 1970-01-01 00:00:01.000000 ./nix/store/fvsswlqh6pc2f5x9qvahm4qbjgjlp86j-linux-pam-1.6.1/lib/
│ │  lrwxrwxrwx   0        0        0       16 1970-01-01 00:00:01.000000 ./nix/store/fvsswlqh6pc2f5x9qvahm4qbjgjlp86j-linux-pam-1.6.1/lib/libpam.so.0 -> libpam.so.0.85.1
│ │  -rwxrwxrwx   0        0        0    67608 1970-01-01 00:00:01.000000 ./nix/store/fvsswlqh6pc2f5x9qvahm4qbjgjlp86j-linux-pam-1.6.1/lib/libpam.so.0.85.1
│ │ --rw-r--r--   0        0        0        0 1970-01-01 00:00:01.000000 ./nix/store/gq7vys7yw9gsrrk5jzp4qx1glc15hwlp-initrd-kmod-blacklist-ubuntu
│ │ +-r--r--r--   0        0        0        0 1970-01-01 00:00:01.000000 ./nix/store/gq7vys7yw9gsrrk5jzp4qx1glc15hwlp-initrd-kmod-blacklist-ubuntu