Skip to content

Comments

uv detector improvements - multi group dev dependencies, git source detection, transitive dev propagation#1631

Open
theztefan wants to merge 2 commits intomicrosoft:mainfrom
theztefan:theztefan/uv-detector-improvements
Open

uv detector improvements - multi group dev dependencies, git source detection, transitive dev propagation#1631
theztefan wants to merge 2 commits intomicrosoft:mainfrom
theztefan:theztefan/uv-detector-improvements

Conversation

@theztefan
Copy link

This pull request enhances the uv detector to address the following limitations identified:

  1. Only the dev group under [package.metadata.requires-dev] was parsed. uv supports arbitrary dependency groups and packages in non-dev groups. These were were silently dropped.

  2. Transitive dependencies of dev-only packages were incorrectly classified as production.

  3. Git-sourced packages were ignore ignored, causing a silent failure to detect them.

Parse all dev dependency groups (Bug fix)

The TOML parser previously hardcoded TryGetValue("dev", ...) to read only the requires-dev table. Now we iterate thru all keys in the requires-dev table collecting dependencies from every group.

Transitive dev dependency propagation (feature)

BFS that computes the full transitive closure from a set of root package names. Dev classification now uses reachability analysis:

  • Compute prodTransitive = all packages reachable from production roots
  • Compute devTransitive = all packages reachable from dev roots
  • devOnly = devTransitive.Except(prodTransitive)

Git source detection (feature)

Packages with source = { git = "..." } are now registered as GitComponent instead of PipComponent. Added ParseGitUrl() to extract the repository URL and commit hash from the URI fragment and registered it. SupportedComponentTypes updated to include the new ComponentType.Git.

And generated tests to cover the new additions.

@theztefan theztefan requested a review from a team as a code owner February 16, 2026 16:08
@microsoft-github-policy-service

@theztefan please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

  1. Definitions.
    “Code” means the computer software code, whether in human-readable or machine-executable form,
    that is delivered by You to Microsoft under this Agreement.
    “Project” means any of the projects owned or managed by Microsoft and offered under a license
    approved by the Open Source Initiative (www.opensource.org).
    “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
    Project, including but not limited to communication on electronic mailing lists, source code control
    systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
    discussing and improving that Project, but excluding communication that is conspicuously marked or
    otherwise designated in writing by You as “Not a Submission.”
    “Submission” means the Code and any other copyrightable material Submitted by You, including any
    associated comments and documentation.
  2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
    Project. This Agreement covers any and all Submissions that You, now or in the future (except as
    described in Section 4 below), Submit to any Project.
  3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
    Should You wish to Submit materials that are not Your original work, You may Submit them separately
    to the Project if You (a) retain all copyright and license information that was in the materials as You
    received them, (b) in the description accompanying Your Submission, include the phrase “Submission
    containing materials of a third party:” followed by the names of the third party and any licenses or other
    restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
    guidelines concerning Submissions.
  4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
    for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
    Submission is made in the course of Your work for an employer or Your employer has intellectual
    property rights in Your Submission by contract or applicable law, You must secure permission from Your
    employer to make the Submission before signing this Agreement. In that case, the term “You” in this
    Agreement will refer to You and the employer collectively. If You change employers in the future and
    desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
    and secure permission from the new employer before Submitting those Submissions.
  5. Licenses.
  • Copyright License. You grant Microsoft, and those who receive the Submission directly or
    indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
    Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
    the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
    parties.
  • Patent License. You grant Microsoft, and those who receive the Submission directly or
    indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
    Your patent claims that are necessarily infringed by the Submission or the combination of the
    Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
    import or otherwise dispose of the Submission alone or with the Project.
  • Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
    No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
    granted by implication, exhaustion, estoppel or otherwise.
  1. Representations and Warranties. You represent that You are legally entitled to grant the above
    licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
    have disclosed under Section 3). You represent that You have secured permission from Your employer to
    make the Submission in cases where Your Submission is made in the course of Your work for Your
    employer or Your employer has intellectual property rights in Your Submission by contract or applicable
    law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
    have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
    You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
    REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
    EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
    PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
    NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
  2. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
    You later become aware that would make Your representations in this Agreement inaccurate in any
    respect.
  3. Information about Submissions. You agree that contributions to Projects and information about
    contributions may be maintained indefinitely and disclosed publicly, including Your name and other
    information that You submit with Your Submission.
  4. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
    the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
    Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
    exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
    defenses of lack of personal jurisdiction and forum non-conveniens.
  5. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the experimental UvLockComponentDetector to better reflect uv.lock semantics by improving dev-dependency parsing/classification and adding support for git-sourced packages.

Changes:

  • Parse all [package.metadata.requires-dev] dependency groups (not just dev).
  • Classify dev-only dependencies using transitive reachability (dev closure minus prod closure).
  • Detect source = { git = "..." } entries and register them as GitComponent (plus new unit tests and updated docs).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Microsoft.ComponentDetection.Detectors/uv/UvLockComponentDetector.cs Adds git URL parsing, dev transitive closure calculation, and registers GitComponent when source.git is present.
src/Microsoft.ComponentDetection.Detectors/uv/UvLock.cs Parses source.git and aggregates dev deps across all requires-dev groups.
src/Microsoft.ComponentDetection.Detectors/uv/UvSource.cs Extends the source model with a Git field.
test/Microsoft.ComponentDetection.Detectors.Tests/UvLockDetectorTests.cs Adds tests for multi-group dev deps, transitive dev propagation, and git source detection.
docs/detectors/uv.md Documents dev reachability behavior and git-sourced component registration.

Comment on lines +42 to +44
var uri = new Uri(gitUrl);
var repoUrl = new Uri(uri.GetLeftPart(UriPartial.Path));
var commitHash = uri.Fragment.TrimStart('#');
Copy link

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ParseGitUrl assumes the git URL is always a valid absolute URI and always contains a non-empty fragment. If the URL is malformed or doesn’t include a commit hash fragment, new Uri(...) or GitComponent construction will throw, causing the detector to bail out for the entire uv.lock. Consider using Uri.TryCreate plus an explicit check for a non-empty commit hash (and falling back to logging + treating it as a non-git package or skipping just that package) to avoid a single bad entry breaking detection.

Suggested change
var uri = new Uri(gitUrl);
var repoUrl = new Uri(uri.GetLeftPart(UriPartial.Path));
var commitHash = uri.Fragment.TrimStart('#');
if (!Uri.TryCreate(gitUrl, UriKind.Absolute, out var uri))
{
// Malformed git URL; signal failure to the caller.
return (null, string.Empty);
}
var repoPart = uri.GetLeftPart(UriPartial.Path);
if (!Uri.TryCreate(repoPart, UriKind.Absolute, out var repoUrl))
{
// Unable to construct a valid repository URL; signal failure.
return (null, string.Empty);
}
var commitHash = uri.Fragment.TrimStart('#');
// If there is no fragment or it is empty/whitespace, treat as "no commit hash".
if (string.IsNullOrWhiteSpace(commitHash))
{
return (repoUrl, string.Empty);
}

Copilot uses AI. Check for mistakes.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is false and not a concern we should take care of at this level. The uv.lock files are machine generated and follow the strict specs which for git sources afaik always includes the commit hash. This means that this could potentially take effect in cases where we have malformed uv.lock but then it should fail because it's malformed 😄

…etector.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 17, 2026 09:26
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Comment on lines 133 to +148
foreach (var dep in pkg.Dependencies)
{
var depPkg = uvLock.Packages.FirstOrDefault(p => p.Name.Equals(dep.Name, StringComparison.OrdinalIgnoreCase));
if (depPkg != null)
{
var depComponentWithVersion = new PipComponent(depPkg.Name, depPkg.Version);
singleFileComponentRecorder.RegisterUsage(new DetectedComponent(depComponentWithVersion), parentComponentId: pipComponent.Id);
TypedComponent depComponent;
if (depPkg.Source?.Git != null)
{
var (depRepoUrl, depCommitHash) = ParseGitUrl(depPkg.Source.Git);
depComponent = new GitComponent(depRepoUrl, depCommitHash);
}
else
{
depComponent = new PipComponent(depPkg.Name, depPkg.Version);
}
singleFileComponentRecorder.RegisterUsage(new DetectedComponent(depComponent), parentComponentId: component.Id, isDevelopmentDependency: isDev);
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependency edge registration does a linear search for every dependency (uvLock.Packages.FirstOrDefault(...)). For larger uv.lock files this becomes O(packages * deps) and you already build a name→package lookup for the transitive-closure calculation. Consider building a single case-insensitive lookup dictionary once (e.g., at the start of OnFileFoundAsync) and reusing it both for transitive reachability and for resolving depPkg when adding edges.

Copilot uses AI. Check for mistakes.
Comment on lines 40 to 124
@@ -49,8 +83,8 @@ protected override Task OnFileFoundAsync(ProcessRequest processRequest, IDiction
var uvLock = UvLock.Parse(file.Stream);

var rootPackage = uvLock.Packages.FirstOrDefault(IsRootPackage);
var explicitPackages = new HashSet<string>();
var devPackages = new HashSet<string>();
var explicitPackages = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var devRootNames = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

if (rootPackage != null)
{
@@ -61,30 +95,57 @@ protected override Task OnFileFoundAsync(ProcessRequest processRequest, IDiction

foreach (var devDep in rootPackage.MetadataRequiresDev)
{
devPackages.Add(devDep.Name);
devRootNames.Add(devDep.Name);
}
}

// Compute dev-only packages via transitive reachability analysis.
// A package is dev-only if it is reachable from dev roots but NOT from production roots.
var prodRoots = rootPackage?.Dependencies.Select(d => d.Name) ?? [];
var prodTransitive = GetTransitivePackages(prodRoots, uvLock.Packages);
var devTransitive = GetTransitivePackages(devRootNames, uvLock.Packages);
var devOnlyPackages = new HashSet<string>(devTransitive.Except(prodTransitive), StringComparer.OrdinalIgnoreCase);

foreach (var pkg in uvLock.Packages)
{
if (IsRootPackage(pkg))
{
continue;
}

var pipComponent = new PipComponent(pkg.Name, pkg.Version);
var isExplicit = explicitPackages.Contains(pkg.Name);
var isDev = devPackages.Contains(pkg.Name);
var detectedComponent = new DetectedComponent(pipComponent);
var isDev = devOnlyPackages.Contains(pkg.Name);

TypedComponent component;
if (pkg.Source?.Git != null)
{
var (repoUrl, commitHash) = ParseGitUrl(pkg.Source.Git);
component = new GitComponent(repoUrl, commitHash);
}
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ParseGitUrl/GitComponent creation can throw (e.g., invalid/relative URL, or missing #<commit> fragment). Because this happens inside the single try/catch around the whole file, one bad git URL will abort processing of the entire uv.lock and result in no components being recorded. Consider using Uri.TryCreate + validating a non-empty (and ideally hex/length-checked) commit hash, and if parsing fails log a warning and either skip just that package or fall back to a PipComponent instead of failing the whole detector run.

Copilot uses AI. Check for mistakes.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comment from above is valid here as well. Don't think we should be handling this here.


// Compute dev-only packages via transitive reachability analysis.
// A package is dev-only if it is reachable from dev roots but NOT from production roots.
var prodRoots = rootPackage?.Dependencies.Select(d => d.Name) ?? [];
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prodRoots is currently derived from rootPackage.Dependencies. However, in this codebase/tests the root package can omit the dependencies array while still declaring production dependencies in [package.metadata].requires-dist (which you already parse into explicitPackages). In that case prodRoots becomes empty, prodTransitive stays empty, and shared transitive dependencies can be incorrectly classified as dev-only. Consider using explicitPackages (or rootPackage.MetadataRequiresDist.Select(d => d.Name)) as the production root set for the reachability analysis instead of rootPackage.Dependencies.

Suggested change
var prodRoots = rootPackage?.Dependencies.Select(d => d.Name) ?? [];
var prodRoots = explicitPackages;

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant