SkillSpector review Static findings need contextual review

A pinned 2.12.0 test of benign, unsafe and warning-only skill text, JSON and SARIF reports, findings exit codes and reviewed baseline suppression.

AgentIndex · Published 2026-11-05 · Updated 2026-11-05

SkillSpector scans AI agent skills before installation for risks such as prompt injection, credential access and supply-chain problems. For teams adopting external skills, it offers a concrete inspection step beyond repository popularity. We evaluated its static mode to establish what it reports without a model and how much interpretation the results require.

Our unsafe instruction produced two findings. A warning that explicitly prohibited the same behavior produced the same two finding categories and risk score. The demonstrated value is therefore a review queue with locations and explanations. A static score alone did not distinguish the intent of these two examples.

Version and inputs

On November 5, 2026, we pinned commit 62c2fe8622f580ac292d4078c3b71f790194dbbd, whose package version is 2.12.0. This is a review of that commit, not a claim to have tested the latest release. The package uses Apache-2.0 licensing. We installed it into an isolated Python 3.12 environment on macOS arm64. Pinned package configuration

Three handwritten SKILL.md inputs described a benign local summary task, an instruction to override existing directions and read a private key, and documentation explaining that those behaviors are prohibited. The unsafe specimen was scanned as data, never installed or executed. Its destination used the reserved domain example.invalid.

Every scan used --no-llm. We made no model call, supplied no dependency manifest and did not test remote ingestion or vulnerability-database lookup. The reproduction fixture includes source fingerprints, installed dependency versions and observed reports.

Input or featureObserved result
Benign local taskNo issues; score 0; exit 0
Deliberately unsafe instructionP1 instruction override and PE3 credential access
Warning prohibiting those behaviorsThe same two finding categories
Risk summary for both flagged inputsScore 42, MEDIUM; highest individual issue HIGH
--fail-on-findingsExit 1 for the flagged inputs
SARIF outputVersion 2.1.0 with two results
Explicit baseline for reviewed warningTwo suppressed, zero active, exit 0

These small synthetic examples provide no statistical estimate of false-positive rates or detection rates on real malicious skills.

Keep the finding evidence

The unsafe specimen's instruction-override phrase triggered P1, while its private-key path triggered PE3. JSON retained rule identifiers, file and line locations, snippets and explanations. Those details allow a reviewer to inspect the original text. Keeping only the aggregate score would discard the information needed to make a decision.

The summary severity was MEDIUM even though the maximum individual severity was HIGH. Before integrating a gate, decide whether policy depends on active findings, individual severity or analysis completeness. We verified that --fail-on-findings changes the exit status. Successful report generation by itself does not establish the absence of findings. CLI options and exit handling

Likewise, a SAFE recommendation in the benign example means the selected scan did not produce issues for that input. We did not exercise every rule, language, archive format or dependency path. No skill ran in a real agent host. Static inspection is separate from runtime isolation and does not enforce installation permissions.

Quotation and negation remain important

The warning specimen said never to follow an instruction override and never to read or transmit credentials. It still received P1 and PE3 findings and the same score as the unsafe instruction. In this tested static path, quotation and negation were not sufficient to distinguish documentation from operational direction.

Security documentation, training material and examples therefore need contextual review. A matched phrase is useful evidence to inspect, but it should not automatically establish malicious intent. Our fixture makes the contrast explicit so that a future version can be checked against the same small case.

The project also offers optional semantic analysis. We did not enable it and cannot claim it resolves this false positive. A separate evaluation would need to record provider configuration, data handling, cost and reproducibility. The reports here record llm_requested: false; that boundary should accompany any comparison. Declared analysis capabilities

Baselines record an acceptance decision

After reviewing the warning as prohibited-behavior documentation, we generated a baseline with --no-llm, recorded a reason and explicitly passed it to another scan of the same directory. Both findings were suppressed, active issues became zero and the findings gate returned zero. This verified suppression; it did not modify the text or repair a defect.

Generating and accepting a baseline for an unknown skill could also hide genuinely unsafe instructions. Review before accepting. The pinned CLI describes author-shipped baselines as opt-in; our actual test used our own explicit baseline and did not exercise that entire discovery path.

For a team trial, retain JSON for contextual review and use SARIF where the receiving platform supports it. Treat failures, incomplete coverage and active findings as separate conditions. We tested the findings gate, not every completeness-control option.

Adoption decision

SkillSpector is worth a limited trial for source review and pre-installation triage when someone will inspect the findings. Pin the skill revision, retain reports and review content changes before extending an accepted baseline. The package requires Python 3.12 or newer. Static mode still installs the normal workflow dependencies; it is not a separate provider-free distribution.

Our results do not establish absolute safety, model-assisted accuracy or automatic intent classification. For choosing resources, continue with the Awesome Copilot review. For a concrete design-skill workflow with browser evidence, see the Taste Skill review.