In reply to @niklaskorz:matrix.org
so how big is the generated index? 😅

Why not do it like this and collect them using nix?

use rayon::iter::ParallelIterator;

#[allow(dead_code)]
#[derive(serde::Serialize)]
struct CrateJson {
    name: String,
    version: String,
    checksum: [u8; 32],
    deps: Vec<String>,
}

fn main() {
    let index = crates_index::GitIndex::with_path("../index", crates_index::git::URL).unwrap();

    println!("Processing crates...");
    index
        .crates_parallel()
        .filter_map(|c| c.ok())
        .for_each(|c| {
            println!("Processing crate: {}", c.name());

            let name = c.name();
            let versions = c.versions();

            let mut major_versions: std::collections::HashMap<u64, Vec<&crates_index::Version>> =
                std::collections::HashMap::new();

            for v in versions {
                if let Ok(semver) = semver::Version::parse(v.version()) {
                    if semver.major == 0 {
                        let key = semver.minor + 1000000;
                        major_versions.entry(key).or_default().push(v);
                    } else {
                        major_versions.entry(semver.major).or_default().push(v);
                    }
                }
            }

            for (_, mut group) in major_versions {
                group.sort_by(|a, b| {
                    let va = semver::Version::parse(a.version()).unwrap();
                    let vb = semver::Version::parse(b.version()).unwrap();
                    vb.cmp(&va)
                });

                if let Some(latest) = group.first() {
                    let semver = semver::Version::parse(latest.version()).unwrap();
                    let major_ver = if semver.major == 0 { 0 } else { semver.major };

                    let prefix = if name.len() >= 2 { &name[..2] } else { name };

                    let dir_path = format!("rust-crates/by-name/{}/{}", prefix, name);
                    std::fs::create_dir_all(&dir_path).unwrap();

                    let file_path = format!("{}/v{}.json", dir_path, major_ver);

                    let crate_json = CrateJson {
                        name: name.to_string(),
                        version: latest.version().to_string(),
                        checksum: *latest.checksum(),
                        deps: latest
                            .dependencies()
                            .iter()
                            .map(|d| d.name().to_string())
                            .collect(),
                    };

                    let json = serde_json::to_string(&crate_json).unwrap();
                    std::fs::write(&file_path, json).unwrap();
                }
            }
        });
}

Another idea I have been ruminating upon is something like:

1. Create a package set like rustCrates, but do not populate it with anything initially
2. Create builders similar to the above idea (add the source, add a symlink builder for cargo-vendor)
3. When someone wants to add a Rust package, a new process is used:
   1. Use a tool that processes lock files, and checks if the required crates are in the package set
   2. If not, it emits the latest semver-compatible to a by-name-like directory tree using the crates.to name
4. Each ~week a program similar to the one I made is ran:
   1. For each crate in the package set, update each semver to the latest-compatible one, should be relatively fast since no downloading is required, the version is in the index, and the required hash is in the index
   2. Write the latest semver-compatible-version and hash in the files
   3. Check the rustsec db as well, and if crates are yanked.
5. For program dependencies in which the semver that is required is already in the pkgset, they do not touch the package set. If they need to wait a ~week for the next version to the bumped to that is fine.