rustPlatform.fetchCargoVendor: de-duplicate git sources by selector

[CathalMullan]

If a Cargo project contains multiple bare git sources (no explicit rev, branch or tag) for the same repo, the lockfile entry looks like:

[[package]]
name = "cosmic-settings-daemon"
version = "0.1.0"
source = "git+https://github.com/pop-os/dbus-settings-bindings#0fa672f8dadb884001ef9a251b149ed432879629"

[[package]]
name = "cosmic-dbus-a11y"
version = "0.1.0"
source = "git+https://github.com/pop-os/dbus-settings-bindings#3b86984332be2c930a3536ab714b843c851fa8ca"

The new vendor code (#387337) treats each unique source string as a separate source:

[source.original-source-git-5]
git = "https://github.com/pop-os/dbus-settings-bindings"
replace-with = "vendored-source-git-5"

[source.original-source-git-7]
git = "https://github.com/pop-os/dbus-settings-bindings"
replace-with = "vendored-source-git-7"

error: source `original-source-git-7` defines source
  https://github.com/pop-os/dbus-settings-bindings,
  but that source is already defined by `original-source-git-5`
note: Sources are not allowed to be defined multiple times.

This change de-duplicates by selector, matching what cargo vendor does.

Encountered this issue while trying to build cosmic-settings-daemon.

cc: @TomaSajt

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

[CathalMullan]

Hmm, looking at this more closely, I don't think this is the right fix if the aim is to match cargo vendor.

cargo vendor deduplicates to:

[source."git+https://github.com/pop-os/dbus-settings-bindings"]
git = "https://github.com/pop-os/dbus-settings-bindings"
replace-with = "vendored-sources"

I'll switch to a draft for now.

[CathalMullan]

Okay, new approach is to de-dupe git sources by selector, which matches upstream cargo vendor behavior.

[Pandapip1]

Probably worth targeting staging-next instead of staging as this is a fix.

[TomaSajt]

Sure, we can automatically deduplicate, but is that what we really want? Why should we give one of the revisions priority over the other? They are not the same.

Also: did this work before #387337 ? I fail to imagine how it could have been broken because of it.

[CathalMullan]

Agree this isn't exactly a great solution, but I don't think there's any other option here.

Latest cosmic-settings-daemon update was merged a month ago and built fine: e0e2946

Only started failing once the new vendor code was merged in from staging.

The old vendor code chose 0fa672f8 as the winning rev:

> git checkout e0e2946b5372d187ad3d60acfc5e5ddebead212c
> nix build .#cosmic-settings-daemon
...
cosmic-settings-daemon>    Compiling cosmic-settings-daemon v0.1.0 (https://github.com/pop-os/dbus-settings-bindings#0fa672f8)
cosmic-settings-daemon>    Compiling cosmic-dbus-a11y v0.1.0 (https://github.com/pop-os/dbus-settings-bindings#0fa672f8)

[TomaSajt]

Okay, I see why it was working before, and why it's not working now.
I thought the package was also a duplicate, but no, it's only the selector that's duplicated.

I really don't see a pretty way of solving this.
Your fix seems to do its job in this case.
However, if we had both a duplicate package and a duplicate selector, even your fix wouldn't work, because we'd be back to the situation that #387337 aimed to solve.

I'll look into whether your approach could be improved.

In the meantime you could patch out one of the revisions in the lockfile over to the other revision. It's not guaranteed to be correct, but it might be fine as a hotfix.

[CathalMullan]

The affected COSMIC packages have been patched already: #502494

AFAIK those were the only 2 packages affected.

[nyabinary]

The affected COSMIC packages have been patched already: #502494

AFAIK those were the only 2 packages affected.

Yeah but this issue is still worth fixing since it could happen to other packages as well.

[thefossguy]

I applied this commit on top of nixos-unstable-small (6c9a78c09ff4d6c21d0319114873508a6ec01655) and built an ISO with COSMIC. I performed the usual testing (that is done when I bump packages) by booting in the ISO and nothing seemed broken there.

I also ran the nixosTests.cosmic* tests.
ALL tests PASS (4/4).

The ISO is on the nix-community x86 builder build01 if anyone wants to verify, at path /nix/store/07c989d20qbywbm4nfa8cmfw57ijvi7f-nixos-cosmic-26.05pre56789.gfedcba-x86_64-linux.iso/iso/nixos-cosmic-26.05pre56789.gfedcba-x86_64-linux.iso.

[thefossguy]

Would love a review from @NixOS/rust btw

[AL-33]

Hi,
I'm getting this error that's blocking me from making changes/updating my system.
Can you tell me how to fix or work around it?

[thefossguy]

@AL-33 what error do you get?

[AL-33]

I was getting this kind of error:

Cargo.toml error: source `original-source-git-7` defines source
  https://github.com/pop-os/dbus-settings-bindings,
  but that source is already defined by `original-source-git-5`
note: Sources are not allowed to be defined multiple times.

when running sudo nixos-rebuild boot --upgrade for the past three days.
It was failing to build 'cosmic-settings-daemon' due to 'dbus-setting-bindings'.

Today, the problem has disappeared, and I was able to update without any
issues.

[Eveeifyeve]

I am not from the rust team, I think this would be a great idea to add this and from code wise I see no issues.

[TomaSajt]

For any new people stumbling across this PR, please do leave a comment so I can see how common this issue is.

[thefossguy]

We encountered this again in bumping the COSMIC DE packages: #527825 (comment)

[nixpkgs-ci]

Backport failed for staging-26.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin staging-26.05
git worktree add -d .worktree/backport-501979-to-staging-26.05 origin/staging-26.05
cd .worktree/backport-501979-to-staging-26.05
git switch --create backport-501979-to-staging-26.05
git cherry-pick -x 55b813b5d77c96dce549930e1449e902839a1bad 100e23324c871a3b9210bd50c59d4f22e6cc452c

[TomaSajt]

Opened backport #528591