The ask was specific:

I want to scan using Semgrep to scan my code in ~/projects/SkillTriage. Show me how to do it step by step. I want to execute the command myself in a separate terminal.

Then, after the first pass:

I want to capture all the steps and screenshots as well so it will be helpful when I write a blog.

That second message changed the job. A command list is easy. A reproducible record — with the failures left in — is the useful artifact.

The two prompts that set the job, and the two results that closed the loop. This is a rendered excerpt, not a screenshot of the app.

What pair coding is for

Pair coding is not “the model writes it and I nod.” It is a division of attention:

• The human owns intent. What are we scanning? What must we not scan? What counts as done? In this session: scan the package source, do not scan the malicious fixtures or the archive, do not execute the target, keep the raw output, and make the screenshots real.
• The agent owns execution and inspection. Write the commands, run them, read the exit code, notice when the output contradicts the claim, fix the command, and document the delta.
• Both own the evidence. A clean scan is a claim. The capture is the proof.

That is the whole method. The rest is what it looked like in practice.

Step 1 — install, then find the binary

Semgrep is an open-source project maintained by Semgrep, Inc.; its source is at github.com/semgrep/semgrep, and the public rules used here come from the Semgrep Registry.

Semgrep was already installed with pipx. What the agent found was a small, ordinary friction: the binary was in ~/.local/bin, which was not on this shell’s PATH.

Semgrep 1.182.1 was already installed; the binary was in ~/.local/bin, not on this shell’s PATH.

That is not a security finding. It is the kind of detail a human would otherwise hit five minutes later and blame on the tool. The agent used the full path, confirmed the version, and moved on. For the rest of this post I write semgrep; if ~/.local/bin is not on your PATH, use the full path.

Step 2 — scan the source, not the whole repo

The first real scan targeted only the package:

semgrep scan \
  --config p/python \
  --config p/secrets \
  --config p/security-audit \
  --metrics=off \
  --disable-version-check \
  src/skilltriage

The package source passed the community packs: 236 rules, 46 files, zero findings. That is a result, not a proof.

The result was clean: 236 rules, 46 files, 0 findings. The temptation is to write “Semgrep found nothing” and move on. That is exactly the wrong move. A scanner’s clean output is only as meaningful as its scope.

So the next question was not “what did it find?” It was “what did it actually scan?”

Step 3 — the clean scan that scanned nothing

The obvious next target was the repo’s malicious fixtures. The scan came back clean:

The first fixture scan looked clean because it scanned nothing. Semgrep ignores tests/ by default; the empty scope is the finding.

Zero findings, zero targets. Semgrep’s default .semgrepignore behaviour excludes test directories, and the fixtures live under tests/. The result was not “safe.” It was “not looked at.”

This is the moment pair coding earns its keep. The agent could have reported the first clean result and stopped. Instead it read the summary, saw “Targets scanned: 0,” and said so. The fix was not to fight the ignore rules. It was to create a small, deliberately unsafe demo outside tests/ and scan that, so the finding screenshot would be real and the scope would be explicit.

Step 4 — what a finding looks like

The demo is twelve lines: subprocess.run(payload, shell=True), eval(payload), and a cleartext HTTP requests.post. It is not part of SkillTriage, and it is never executed.

A deliberately bad twelve-line demo, used only to show what a finding looks like: shell=True, eval, and cleartext HTTP. The demo is never executed.

Three blocking findings, each with a rule ID, a file and line, a rationale, and a fix hint. This is the part that is easy to demonstrate and easy to misunderstand: the point is not that Semgrep can find eval. The point is that the same command, with the same rules, produced a different result on a different scope. The evidence is the pair of runs, not either one alone.

Step 5 — write a rule, break it, fix it

SkillTriage has a design invariant: it must not start processes, open sockets, or evaluate strings. Its unit tests enforce that. Semgrep can enforce it at scan time too, with two project-specific rules:

• skilltriage-no-process-or-network
• skilltriage-no-dynamic-execution

The first draft used from subprocess import .... Semgrep rejected it:

The first custom rule was rejected: from subprocess import ... is not a valid Semgrep pattern. Changing ... to $FUNC made it valid.

The error was precise: Invalid pattern for Python, Stdlib.Parsing.Parse_error. The fix was small: use $FUNC for the imported name. The rule validated, and the scan passed:

Two project-specific invariants — no process/network imports, no dynamic execution — pass on 45 files.

This is the most honest screenshot in the set. It records a failed attempt and the correction, not just the green result. A pair-coding session that hides its first draft is a demo, not a record.

Step 6 — the gate is a policy choice

The same scan can pass or fail depending on what you decide to gate on. With --error --severity ERROR --severity WARNING:

Same command, two targets. The package exits 0; the demo exits 1. The gate filtered to ERROR/WARNING, so the INFO-level HTTP finding did not fail the build.

The source exits 0. The demo exits 1. But note the count: the full demo scan had three findings; the gate shows two. The cleartext-HTTP finding is INFO-level and was filtered out by the severity choice. It is still in the full report. That is not a bug; it is a policy decision, and it should be visible.

For CI, the same run can produce SARIF:

SARIF 2.1.0 with three results, ready for GitHub code scanning.

The SARIF file is the handoff point. GitHub code scanning can consume it; another tool can join on the rule IDs and locations. The terminal output is for the human. The SARIF is for the pipeline.

The friction we kept

Two things went wrong that are worth recording.

Semgrep was installed but not on PATH. The fix was to call ~/.local/bin/semgrep directly. Boring, real, and the kind of thing that makes a “quick start” fail.

Semgrep 1.182.1 intermittently aborted with io_uring_queue_init: Cannot allocate memory. This was an environment/runtime issue, not a rule issue. The workaround was:

export EIO_BACKEND=posix
semgrep scan --jobs 1 ...

With the POSIX backend and one job, the runs were stable. The screenshots show the commands without the workaround, for readability; the captures used it. On a normal desktop you probably will not need it. If you do see that error, now you know where to look.

Neither of these belongs in a polished “getting started” box. Both belong in a pair-coding record, because the next person will hit them.

What this does not prove

Semgrep found no issues in the SkillTriage package under the community packs and the two project rules. That does not mean SkillTriage is secure. It means the configured rules did not trigger on the files that were scanned.

What is out of scope here:

• No dynamic testing. Nothing was executed. The demo file was read, not run.
• No auto-fix. We did not let Semgrep rewrite code. The shell=True finding is a demonstration, not a request to patch.
• No CI change. The SARIF is ready, but the workflow change is a separate decision. Semgrep’s public rules also change over time; a stable gate should pin a version or vendor the rules.
• No Semgrep App upload. --metrics=off and semgrep scan keep the run local. The registry rules are downloaded; the code is not uploaded.
• No scan of the fixtures or archive by default. They are deliberately hostile or superseded. Scanning them produces noise, and the empty-scope lesson above is why we check the target count before trusting a result.

Semgrep is complementary to SkillTriage’s own 107 rules, the unit suite, Bandit, and pip-audit. It is another lens, not a verdict.

What pair coding actually looks like

The loop that made the post possible: the human sets intent and constraints; the agent runs, reads, and reports; both check the same output.

The useful version of “vibe coding” is not prompt-and-pray. It is a loop with a paper trail:

1. The human states intent and constraints.
2. The agent proposes the smallest command that can test the claim.
3. The agent runs it and reads the scope, the count, and the exit code.
4. The human checks whether the result answers the question that was asked.
5. When something is wrong, the agent fixes the command or the rule and re-runs the same check.
6. Both keep the output, the failure, and the limit.

The artifacts are not just the code. They are the runbook, the raw captures, the screenshots, the version, the exit codes, and the list of what was not tested. That is what makes the session reusable by someone else — or by me, six months from now.

SkillTriage is at github.com/cicada-369/SkillTriage. Semgrep’s source is at github.com/semgrep/semgrep; the public rules are in the Semgrep Registry. If you try the same loop, the most useful thing you can send back is the place where the output and the claim disagreed.

Bo Kaung · Builder/Breaker/Defender · LinkedIn · Hashnode

I was working through the OWASP Top 10 lists when I found the OWASP Agentic Skills Top 10. It names exactly what I had been ignoring:

1. Malicious Skills
2. Supply Chain Compromise
3. Over-Privileged Skills
4. Insecure Metadata
5. Untrusted External Instructions
6. Weak Isolation
7. Update Drift
8. Poor Scanning
9. No Governance
10. Cross-Platform Reuse

I had never validated my own skills and plugins. Through 2026, organizations including Cyber Security News, the Cloud Security Alliance, and the Non-Human Identity Management Group have documented the same pattern: agent capabilities are spreading faster than the review around them. A skill runs with the agent's privileges. A plugin can register a hook that runs automatically. A remote MCP server sees what the agent sends it.

So I built the tool I wanted: a static auditor that inspects a skill or plugin before installation, shows evidence, and never executes anything. I called it SkillTriage — triage, not a verdict.

A failing scan: download-and-execute, an instruction-manipulation chain, and an approval bypass.

What it is

SkillTriage is a Python 3.11+ CLI with zero third-party runtime dependencies. It reads a skill directory, a Claude Code plugin root, or a .zip/.tar.gz, runs 107 rules mapped to AST01–AST10, and emits text, JSON, Markdown, SARIF, or a self-contained offline HTML report. It never executes target code, fetches a URL, or calls a model — the property that makes it safe to point at something you do not trust.

A plugin is treated as one artifact: manifest, commands, agents, bundled skills, hooks, MCP servers, and marketplace entry are inspected together. The riskiest part is often not a SKILL.md; it is the hook that runs on every tool call, or the MCP server the manifest starts.

What it looks like

The HTML report is one file: no CDN, fonts, external images, or network requests; a restrictive CSP; owner-only permissions by default, because reports can contain excerpts from the artifact.

The HTML report is one offline file: summary, evidence, rationale, remediation, and CWE references.

The terminal output is the same data with fewer pixels: what failed, why, where the evidence is, and how to fix it.

What it catches

• Execution and manipulation. Download-and-execute, reverse shells, destructive commands, persistence, credential access, exfiltration, identity writes, prompt overrides, concealment, and approval bypasses.
• Supply chain. Unpinned or URL/VCS dependencies, install scripts, executable configuration, missing provenance, unpinned MCP launches, and cleartext MCP transport.
• Over-privileged components. Shell, filesystem, network, and tool scope; wildcard grants; plugin commands and agents asking for more than the task. A scoped Bash(ls *) is not broad; an unscoped Bash(*) is.
• Metadata and hooks. Unsafe YAML, duplicate keys, invisible characters, escaping or missing component paths; PreToolUse/PermissionRequest hooks that return an allow decision, escape the plugin root, or reference a missing script.
• MCP and secrets. npx pkg@latest, uvx --from git+…, images without a digest, non-loopback http://, literal header/env credentials (values are never stored), PEM and URL secrets, base64 payloads, invisible Unicode.
• Context, not suppression. Docs and examples lower noise, but outside a documentation path a critical finding cannot be talked down by a comment or a word such as “test”. Declared intent can downgrade technique rules only with an explicit policy allowance and a matching inventory hash.

How it works

Five stages, no execution and no network. The report formats are rendered only at the end.

1. Load safely. Walk without following symlinks, open with O_NOFOLLOW, bound every read, and defend archive extraction against zip-slip, bombs, traversal, links, devices, and decompression.
2. Parse safely. A restricted YAML subset, json and tomllib, with explicit nesting bounds and no object construction.
3. Normalize. Skills and plugin artifacts become one model.
4. Run 107 rules. Score context and correlate credential + network, or manipulation + execution, into higher-severity chains. Correlations use code, SKILL.md, or structured plugin config — never command or agent prose.
5. Gate and report. Apply policy, suppressions, baseline, and declared intent; render text, JSON, Markdown, SARIF, or HTML.

The rule count is a map of coverage, not a quality score:

Rule distribution by OWASP control at the time of writing. Counts show coverage, not quality.

What I found on my machine

I ran it over my own installed corpus: 67 targets — 43 Claude Code plugins and 24 standalone skills — with zero scan errors and zero rule errors. Only one target gated: an offensive-security skill I wrote myself. That is the case the intent policy exists for.

The rest produced a handful of genuine findings — four unpinned MCP launches and some low-severity duplicate component names — but no plugin gated under the default policy. That is a useful result, not proof that those plugins are safe. It means the configured checks did not find enough evidence to block.

The part I care about most: not making things worse

A tool that reads hostile input has to survive hostile input. The loader refuses a symlinked target root, never follows a symlink inside a target, bounds per-file reads, and treats archive decompression and entry counts as attack surfaces. Parsers cannot construct objects. Every report field is escaped for control, bidi, and invisible characters before it reaches a terminal or HTML.

Two decisions came out of an independent review:

• Inline suppressions are opt-in. A skilltriage:disable comment lives inside the artifact, so honoring it by default would let a package silence its own findings. Baselines are the reviewed-acceptance mechanism; inline suppressions require --trust-inline-suppressions.
• A target cannot weaken its own gate. Project config is not auto-loaded from the current directory; it must be passed with --config. A declared intent is honored only when policy allows it and the inventory record's content hash matches the current artifact. Findings are downgraded, never removed.

The reviewer found three critical issues and a set of high and medium ones. The critical issues were: a skill could suppress every finding against itself with a comment, a planted project config could disable the gate, and a one-word context trick could push a critical payload below the threshold. All three are fixed with regression tests; the high and medium findings are fixed or documented as limits. The full review, including what was not tested, is in docs/security/review-v0.1.md.

What it does not do

• No dynamic testing. It does not execute hooks, start MCP servers, or observe runtime file and network activity. That is the sandboxed v0.2 milestone, with its own threat model.
• Hooks and MCP are inspected, not run. Approval-injection detection follows the hook command and the script it directly references; an indirectly invoked script is a gap.
• No signature verification. Signature presence is recorded, not verified. Dependencies are inspected, not resolved, so transitive CVEs are out of scope. URLs are classified, not fetched.
• Marketplace scope. A marketplace manifest is evaluated only at an audited plugin root; standalone Connectors are not audited.
• False positives happen around security documentation, red-team tooling, and detector code. Context scoring, baselines, and intent are the controls. A clean report means the configured checks did not trigger; it does not prove safety.
• Platform coverage. The suite runs on Linux today; Windows and macOS file modes and archive semantics have not been exercised.

Try it

SkillTriage is Apache-2.0 and lives at github.com/cicada-369/SkillTriage. It is not on PyPI in v0.1; install from source or use the single-file zipapp.

git clone https://github.com/cicada-369/SkillTriage.git
cd SkillTriage
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

# Scan the default install locations
skilltriage scan

# One plugin, with a shareable offline report
skilltriage scan ~/.claude/plugins/cache/<market>/<plugin>/<version> \
  -o report.html --open

# A downloaded package, as SARIF for CI
skilltriage audit ./downloaded-skill --format sarif -o skilltriage.sarif

The default gate fails on HIGH or CRITICAL findings with HIGH confidence. --profile advisory never fails; --profile strict gates at MEDIUM. For CI, SARIF works with GitHub code scanning, and --fail-on none makes a scan report-only.

If you maintain an offensive-security skill, declare its intent and inventory it before allowing the downgrade:

# metadata.skilltriage.intent: offensive-security in SKILL.md
skilltriage scan --inventory --update-inventory
skilltriage scan --inventory --allow-intent offensive-security

The declaration alone changes nothing: the policy must allow the value and the inventory hash must match the bytes you reviewed. The technique findings stay in every report, with the downgrade visible.

More detail: plugin coverage, declared intent, and the rule catalog.

What’s next

The next layer is the one the tool refuses to fake: sandboxed dynamic testing. Start the MCP server, run the hook, capture tool calls, and see what the artifact actually does — behind explicit consent, network-none defaults, and syscall/process observation. Static analysis tells you what to review; a sandbox tells you what happened. After that: signature and content-hash verification, SBOM/CVE resolution, and more plugin ecosystems.

If you try SkillTriage, the most useful thing you can send me is what it gets wrong. False positives make a scanner easy to ignore; false negatives are worse. Open an issue, or use the private security advisory flow for vulnerabilities.

Your agent's skills are code. Audit them before you trust them.

Bo Kaung · Builder/Breaker/Defender · LinkedIn · Hashnode

Further reading: the OWASP Agentic Skills Top 10 and its published site; SkillTriage; the SARIF specification and GitHub code scanning.