| 17 May 2023 |
 hexa | in a bit 😄 | 19:24:37 |
| hexa | it's an unlanded PR rebased and locally updated to a newer version during rebase 🤡 | 19:24:53 |
 Lily Foster | (alternatively I can just go figure out why npm gets too spooked to use the cache with workspaces like this -- their lack of instrumentation/debug messaging really makes it difficult to track down where decisions like "oh the cache is bad, time to refetch" are made) | 19:25:14 |
| hexa | there is the django-hijack issue above, if you want to knock yourself out 😄 | 19:25:30 |
| Lily Foster | (especially since it doesn't log that it even decides the cache is bad) | 19:25:39 |
| Lily Foster | In reply to @hexa:lossy.network
there is the django-hijack issue above, if you want to knock yourself out 😄 Oh yeah I need to come back to that one. I'll do that right now | 19:25:48 |
| Lily Foster | In reply to @hexa:lossy.network
node_modules/webpack/bin/webpack.js: a /nix/store/6aphgr19pzc8v416zaf6vbs7r9jsbajp-nodejs-18.16.0/bin/node script, ASCII text executable
So this is right. But the setup.py also runs npm ci right before the build script which undoes the fixing that the hook already did (ci is for clean-installing deps, but the deps should already be installed) | 19:56:58 |
| Lily Foster | This diff to your derivation makes it work:
diff --git a/pkgs/development/python-modules/django-hijack/default.nix b/pkgs/development/python-modules/django-hijack/default.nix
index ff0eb44ef44..34457b923c6 100644
--- a/pkgs/development/python-modules/django-hijack/default.nix
+++ b/pkgs/development/python-modules/django-hijack/default.nix
@@ -32,6 +32,7 @@ buildPythonPackage rec {
postPatch = ''
substituteInPlace setup.py \
+ --replace 'cmd = ["npm", "ci"]' 'cmd = ["true"]' \
--replace 'f"{self.build_lib}/{name}.mo"' 'f"{name}.mo"'
'';
| 19:57:08 |
| hexa | ok | 19:58:35 |
| hexa | cool | 19:58:37 |
| Lily Foster | (if you --keep-failed and check the webpack.js file after the build fails, you'll notice the shebang goes back to #!/usr/bin/env node) | 19:59:26 |
| Lily Foster | (despite being correct at preBuild) | 19:59:37 |
| hexa | thanks! | 20:07:34 |
| Lily Foster | For once it wasn't npm's fault, surprisingly | 20:07:55 |
| Lily Foster | In reply to @hexa:lossy.network
it's an unlanded PR rebased and locally updated to a newer version during rebase 🤡 If you want to test it yourself, add a npmWorkspace argument with the npm workspace package you are wanting to build and apply this diff locally:
diff --git a/pkgs/build-support/node/build-npm-package/default.nix b/pkgs/build-support/node/build-npm-package/default.nix
index 1c3fb6a74ef..0aa5ade9dc3 100644
--- a/pkgs/build-support/node/build-npm-package/default.nix
+++ b/pkgs/build-support/node/build-npm-package/default.nix
@@ -30,6 +30,8 @@
, npmBuildFlags ? [ ]
# Flags to pass to `npm pack`.
, npmPackFlags ? [ ]
+ # Value for npm `--workspace` flag and directory in which the files to be installed are found.
+, npmWorkspace ? null
, ...
} @ args:
@@ -45,6 +47,8 @@ in
stdenv.mkDerivation (args // {
inherit npmDeps npmBuildScript;
+ npmFlags = args.npmFlags ++ lib.optionals (npmWorkspace != null) [ "--workspace" npmWorkspace ];
+
nativeBuildInputs = nativeBuildInputs ++ [ nodejs npmConfigHook npmBuildHook npmInstallHook ];
buildInputs = buildInputs ++ [ nodejs ];
diff --git a/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh b/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
index 59ea2da993d..afc911cd021 100644
--- a/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
+++ b/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
@@ -13,7 +13,7 @@ npmInstallHook() {
while IFS= read -r file; do
local dest="$packageOut/$(dirname "$file")"
mkdir -p "$dest"
- cp "$file" "$dest"
+ cp "${npmWorkspace:-.}/$file" "$dest"
done < <(@jq@ --raw-output '.[0].files | map(.path) | join("\n")' <<< "$(npm pack --json --dry-run $npmPackFlags "${npmPackFlagsArray[@]}" $npmFlags "${npmFlagsArray[@]}")")
while IFS=" " read -ra bin; do
@@ -22,7 +22,7 @@ npmInstallHook() {
done < <(@jq@ --raw-output '(.bin | type) as $typ | if $typ == "string" then
.name + " " + .bin
elif $typ == "object" then .bin | to_entries | map(.key + " " + .value) | join("\n")
- else "invalid type " + $typ | halt_error end' package.json)
+ else "invalid type " + $typ | halt_error end' "${npmWorkspace:-.}/package.json")
local -r nodeModulesPath="$packageOut/node_modules"
| 20:17:45 |
| Lily Foster | Otherwise I'll wait for a derivation to poke at | 20:17:52 |
| hexa | working on a reverse dependency | 20:18:10 |
| hexa | #soon | 20:18:15 |
| Lily Foster | In reply to @hexa:lossy.network
it's an unlanded PR rebased and locally updated to a newer version during rebase 🤡 * If you want to test it yourself, add a npmWorkspace argument with the npm workspace package you are wanting to build and apply this diff locally:
diff --git a/pkgs/build-support/node/build-npm-package/default.nix b/pkgs/build-support/node/build-npm-package/default.nix
index 1c3fb6a74ef..0aa5ade9dc3 100644
--- a/pkgs/build-support/node/build-npm-package/default.nix
+++ b/pkgs/build-support/node/build-npm-package/default.nix
@@ -30,6 +30,8 @@
, npmBuildFlags ? [ ]
# Flags to pass to `npm pack`.
, npmPackFlags ? [ ]
+ # Value for npm `--workspace` flag and directory in which the files to be installed are found.
+, npmWorkspace ? null
, ...
} @ args:
@@ -45,6 +47,8 @@ in
stdenv.mkDerivation (args // {
inherit npmDeps npmBuildScript;
+ npmFlags = npmFlags ++ lib.optionals (npmWorkspace != null) [ "--workspace" npmWorkspace ];
+
nativeBuildInputs = nativeBuildInputs ++ [ nodejs npmConfigHook npmBuildHook npmInstallHook ];
buildInputs = buildInputs ++ [ nodejs ];
diff --git a/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh b/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
index 59ea2da993d..afc911cd021 100644
--- a/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
+++ b/pkgs/build-support/node/build-npm-package/hooks/npm-install-hook.sh
@@ -13,7 +13,7 @@ npmInstallHook() {
while IFS= read -r file; do
local dest="$packageOut/$(dirname "$file")"
mkdir -p "$dest"
- cp "$file" "$dest"
+ cp "${npmWorkspace:-.}/$file" "$dest"
done < <(@jq@ --raw-output '.[0].files | map(.path) | join("\n")' <<< "$(npm pack --json --dry-run $npmPackFlags "${npmPackFlagsArray[@]}" $npmFlags "${npmFlagsArray[@]}")")
while IFS=" " read -ra bin; do
@@ -22,7 +22,7 @@ npmInstallHook() {
done < <(@jq@ --raw-output '(.bin | type) as $typ | if $typ == "string" then
.name + " " + .bin
elif $typ == "object" then .bin | to_entries | map(.key + " " + .value) | join("\n")
- else "invalid type " + $typ | halt_error end' package.json)
+ else "invalid type " + $typ | halt_error end' "${npmWorkspace:-.}/package.json")
local -r nodeModulesPath="$packageOut/node_modules"
| 20:21:49 |
| hexa | Lily Foster: https://github.com/NixOS/nixpkgs/pull/215406 | 20:38:29 |
| hexa | disregard the panel build, this is about bokeh | 20:38:37 |
| Lily Foster | What is this bespoke build system that bokehjs made... | 21:14:56 |
| Lily Foster | Yeah I'm gonna have to figure out the underlying issue of why npm workspaces cause npm to not use the cache for this one | 21:15:58 |
| hexa | 😄 | 21:16:50 |
| hexa | yeah, it was pure suffering to get 3.0.3 source build going | 21:17:01 |
| hexa | https://github.com/NixOS/nixpkgs/pull/229953 trying out the npmRoot thing | 21:19:55 |
| hexa |
navidrome> ERROR: Missing package-lock.json from src. Expected to find it at: /build/source/source/ui/package-lock.json
| 21:20:09 |
| hexa | good error message | 21:20:12 |
| hexa | but once I fixed the path it went poof | 21:20:28 |
| hexa | navidrome> unpacking sources
navidrome> unpacking source archive /nix/store/f51j4vbggdg3a7bq8xr8ksf3a5xs0wvv-source
navidrome> source root is source
navidrome> patching sources
navidrome> Executing npmConfigHook
navidrome> Configuring npm
navidrome> Validating consistency between /build/source/ui/package-lock.json and /nix/store/y2splh2dkp37gjkn77hqkscbsd9x89cj-npm-deps/package-lock.json
navidrome> Fixing lockfile
navidrome> Installing dependencies
navidrome> npm ERR! code EUSAGE
navidrome> npm ERR!
navidrome> npm ERR! The `npm ci` command can only install with an existing package-lock.json or
navidrome> npm ERR! npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm@5 or
navidrome> npm ERR! later to generate a package-lock.json file, then try again.
navidrome> npm ERR!
navidrome> npm ERR! Clean install a project
navidrome> npm ERR!
navidrome> npm ERR! Usage:
navidrome> npm ERR! npm ci
navidrome> npm ERR!
navidrome> npm ERR! Options:
navidrome> npm ERR! [-S|--save|--no-save|--save-prod|--save-dev|--save-optional|--save-peer|--save-bundle]
navidrome> npm ERR! [-E|--save-exact] [-g|--global]
navidrome> npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
navidrome> npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
navidrome> npm ERR! [--strict-peer-deps] [--no-package-lock] [--foreground-scripts]
navidrome> npm ERR! [--ignore-scripts] [--no-audit] [--no-bin-links] [--no-fund] [--dry-run]
navidrome> npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
navidrome> npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
navidrome> npm ERR!
navidrome> npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
navidrome> npm ERR!
navidrome> npm ERR! Run "npm help ci" for more info
navidrome>
navidrome> npm ERR! Log files were not written due to an error writing to the directory: /nix/store/y2splh2dkp37gjkn77hqkscbsd9x89cj-npm-deps/_logs
navidrome> npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal
navidrome>
navidrome> ERROR: npm failed to install dependencies
navidrome>
navidrome> Here are a few things you can try, depending on the error:
navidrome> 1. Set `makeCacheWritable = true`
navidrome> Note that this won't help if npm is complaining about not being able to write to the logs directory -- look above that for the actual error.
navidrome> 2. Set `npmFlags = [ "--legacy-peer-deps" ]`
navidrome>
| 21:20:32 |
