Files
github-spec-kit/tests/extensions/assess/test_assess_extension.py
Manfred Riem 208d38695f feat(extensions): add assess idea assessment pipeline extension (#3568)
* feat(extensions): add assess idea assessment pipeline extension

Add a role-neutral, opt-in "Idea Assessment Pipeline" extension (id:
assess) covering the discovery work that happens BEFORE spec-driven
development. It provides a five-stage funnel: intake, research, define,
shape, decide, each writing one artifact under
.specify/assessments/<slug>/. A go verdict hands off to
/speckit.specify; killing an idea is a first-class success outcome.

Registration:
- extensions/catalog.json: bundled core opt-in entry (before bug)
- pyproject.toml: force-include maps into core_pack so it ships in the
  installed wheel (verified via wheel build)

Also normalizes a Rich-wrapped substring assertion in test_workflows.py
so the suite passes at CI's 80-column non-TTY width.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): address PR review on assess extension

Resolve review feedback on github/spec-kit#3568:

- catalog.json: bump top-level updated_at to this revision (2026-07-17)
- extension.yml + catalog.json: shorten the assess description to under
  the documented 200-char manifest limit (kept aligned across both)
- extension.yml: make the before_specify hook prompt condition-neutral
  (it fires on every /speckit.specify, so it must not claim "no
  assessment found")
- intake.md: fix slug normalization to explicitly allow lowercase
  letters a-z (the old rule permitted only digits and '-', contradicting
  the offline-mode example)
- intake.md + research.md: require a sanitized source URL (strip
  userinfo and credential/signature query params) instead of persisting
  a verbatim URL that could leak secrets into project artifacts
- decide.md: remove the "trivially small" exception so a go always
  requires a shaped concept, making verdict behavior deterministic and
  consistent with the guardrails and README

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* refactor(extensions): remove before_specify hook from assess

Assess is a separate business process from spec-driven development, so
it should not inject itself into the /speckit.specify lifecycle. The
hook fired on every /speckit.specify invocation (it had no condition),
nagging even when an assessment already existed and the user was
deliberately proceeding.

Unlike git's before_specify (a mechanical prerequisite: create a feature
branch) or agent-context's after_* hooks (reacting to spec output),
assess is an upstream, optional, human-judgment process. The coupling
that belongs here already runs forward and by choice: a `go` verdict
from /speckit.assess.decide hands off to /speckit.specify. The backward
hook was the redundant, intrusive direction.

- extension.yml: drop the hooks block (commands-only manifest)
- README.md: replace the Hooks section with a Handoff section
- test: replace the hook assertion with test_declares_no_hooks to lock
  in the standalone-pipeline design

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): harden assess slug handling and clarify verdict logic

Address the second review round on github/spec-kit#3568:

- Slug path traversal: intake and all four downstream commands
  (research, define, shape, decide) now normalize an explicit or
  user-supplied slug to the [a-z0-9-] alphabet (dropping '.', '/', '\\')
  and reject an empty normalized result before constructing ASSESS_DIR.
  This guarantees a slug like `../..` cannot escape .specify/assessments/.
- Metadata accuracy: the extension.yml and catalog.json descriptions no
  longer imply a "build/kill" call is handed to /speckit.specify — only a
  `go` hands off; a `kill` closes the assessment.
- Verdict determinism (decide): a `go` now explicitly requires evidence
  strength `adequate`+ (never weak/unknown), resolving the conflict with
  the thin-evidence guardrail.
- Risk polarity (decide): renamed the "Risk" criterion to "Risk posture"
  with positive polarity (strong = risks understood and mitigated) so it
  composes with the other scores that feed the verdict.
- README: aligned the go-threshold guardrail with the evidence rule and
  documented the slug-normalization safety property.

The PR description was also updated to drop the stale before_specify
hook claim (the hook was removed in the previous commit).

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): add symlink/realpath containment and pin research host allowlist

Address the third review round on github/spec-kit#3568:

- Path safety (intake, research, define, shape, decide): slug
  normalization blocks lexical `..` but not symlinked path components.
  Each command now, before any mkdir/read/write, resolves the real path
  of .specify/assessments/<slug>/ and every artifact, refuses to follow a
  symlinked .specify / assessments / slug dir / artifact, and verifies the
  resolved path stays inside the project root. This blocks a cloned or
  crafted project from redirecting reads/writes outside the repository.
  Each stage enforces this independently since research/define/decide can
  run without intake.
- research URL policy: replaced the open-ended "and comparable well-known
  hosts" no-prompt branch with intake's exact enumerated allowlist, so an
  agent cannot classify an attacker-controlled host as "comparable" and
  fetch it without confirmation.
- README: guardrail now documents symlink/realpath containment.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): redact secrets in captured idea and stop on explicit-slug collision

Address the fourth review round on github/spec-kit#3568 (intake):

- Secret leak in the captured idea: quoting the original "verbatim"
  contradicted the URL sanitization rule when the idea itself contained a
  credential-bearing URL. Capture now redacts secrets (sanitize URLs;
  strip tokens, passwords, keys, cookies) inside the quoted text as well
  as the Source field, and the section heading is "Idea (as captured)"
  rather than "verbatim".
- Explicit-slug collision: in automated mode an existing intake.md caused
  a silent switch to a new slug, contradicting the no-suffix guarantee for
  user-provided slugs. Now: user-provided slug collision -> stop and
  report; only a self-generated slug (already disambiguated at resolution)
  is re-slugged.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): reject IPv6 private ranges and DNS-rebinding in URL policy

Address the remaining open comment from review 4722852090 on
github/spec-kit#3568 (the other six comments in that round were already
resolved by the slug-validation and host-allowlist fixes in 9cd07fb and
c032a2e).

The URL Trust Policy refused only textual IPv4 loopback/RFC1918/metadata
hosts, so an approved hostname resolving to an internal IPv6 or
IPv4-mapped address could still reach internal services. The refuse-
outright list now covers IPv6 link-local (fe80::/10), unique-local
(fc00::/7), IPv4-mapped forms, and the IPv6 metadata address, and adds a
resolution-time check: even an allowlisted or user-confirmed host is
refused when it resolves to any non-public address, defeating DNS
rebinding. Mirrored the summary in research's inherited-policy note.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): pin connection vs DNS rebinding and gate slug-only direct entry

Address the fifth review round on github/spec-kit#3568:

- DNS rebinding (intake + research): a standalone DNS lookup does not
  defeat rebinding because the fetch client can re-resolve or pick a
  private address from a mixed answer. The policy now requires the fetch
  to pin the connection to a validated public address (or verify the
  connected peer) and re-apply the refusal ranges to the address actually
  connected to; if the fetch mechanism cannot pin or expose the peer, the
  fetch is refused rather than trusted by hostname.
- Slug-only direct entry (research + define): when intake/research
  artifacts are absent and $ARGUMENTS carries only a slug, the commands no
  longer infer an idea/problem from the slug. They now require substantive
  idea/problem text and otherwise prompt (interactive) or stop (automated).
- Cleaned up a leftover duplicate ASSESS_DIR assignment line in research.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* fix(extensions): allow read-only source inspection in intake guardrail

Address the sixth review round on github/spec-kit#3568.

The intake guardrail said the command "only reads and writes inside
.specify/assessments/<slug>/", which contradicts its documented inputs:
intake must read a codebase pointer (repository inspection) and fetch an
allowed URL to capture the idea. The guardrail now limits only *writes*
to the assessment directory and explicitly permits read-only inspection
of the supplied sources (repo + allowlisted URL fetch under the URL Trust
Policy). The other four commands already phrased this correctly ("read
only, and write inside ...") and are unchanged.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* Harden assess commands: ancestor path safety, untrusted-artifact reads, research dir creation

Addresses review 4723905370 on PR #3568 across three themes:

- Ancestor path safety: verify `.specify` and `.specify/assessments` are
  real directories (not symlinks) resolving inside the project root before
  any filesystem-based slug resolution, in all five commands.
- Untrusted artifact reads: treat the contents of persisted assessment
  artifacts (intake/research/problem/concept) as untrusted data, not
  instructions — ignore embedded directives, mirroring the URL Trust Policy.
- research now ensures the validated ASSESS_DIR exists before writing, since
  it may be the first assessment command run.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* Allow absent assessment dir in ancestor path-safety check

Addresses review 4723955260 on PR #3568. The ancestor path-safety clause
required `.specify/assessments` to already be a real directory, which blocked
the first-run commands (intake, research, define) from ever reaching the step
that creates it. Reword the clause in all five commands so a not-yet-created
directory is permitted, while still refusing when `.specify` or
`.specify/assessments` exists as a symlink or escapes the project root.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* Fix README diagram: needs-clarification revisits the named earlier stage

Addresses review 4724027270 on PR #3568. The overview flowchart routed every
needs-clarification verdict back to research, but decide.md's Revisit stage can
send an idea back to intake, research, define, or shape. Reroute the arrow as a
generic loop back to the earlier stages so the diagram no longer misstates the
pipeline.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

* Make decide handoff integration-neutral (no hard-coded dot-style)

Addresses review 4724080304 on PR #3568:

- decide.md frontmatter description hard-coded `/speckit.specify`. Frontmatter
  is parsed before command-reference resolution, so it now uses agent-neutral
  wording ("hand survivors off into Spec-Driven Development") instead of a
  dot-style literal that would be wrong for non-dot integrations.
- The `## If go — Handoff to …` heading inside the decision.md output template
  hard-coded `/speckit.specify`, which would be written verbatim into
  decision.md. It now uses the `__SPECKIT_COMMAND_SPECIFY__` placeholder, like
  the rest of the command, so the active integration's invocation style is
  rendered.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, supervised)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1f25cf9d-b7eb-4b2b-b811-3e91d8db8f6a

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-07-17 11:11:14 -05:00

125 lines
4.7 KiB
Python

"""Tests for the bundled ``assess`` extension.
Validates:
- Bundled layout (manifest, README, five command files)
- Catalog registration
- Wheel/source-checkout resolution via ``_locate_bundled_extension``
- Install via ``ExtensionManager.install_from_directory`` copies the five
command files and records them in the installed manifest (command
registration with AI agents is exercised separately and not asserted here)
"""
from __future__ import annotations
import json
from pathlib import Path
import yaml
from specify_cli import _locate_bundled_extension
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent.parent
EXT_DIR = PROJECT_ROOT / "extensions" / "assess"
EXPECTED_COMMANDS = {
"speckit.assess.intake",
"speckit.assess.research",
"speckit.assess.define",
"speckit.assess.shape",
"speckit.assess.decide",
}
# ── Bundled extension layout ─────────────────────────────────────────────────
class TestExtensionLayout:
def test_extension_yml_exists(self):
assert (EXT_DIR / "extension.yml").is_file()
def test_extension_yml_has_required_fields(self):
manifest = yaml.safe_load(
(EXT_DIR / "extension.yml").read_text(encoding="utf-8")
)
assert manifest["extension"]["id"] == "assess"
assert manifest["extension"]["name"] == "Idea Assessment Pipeline"
assert manifest["extension"]["author"] == "spec-kit-core"
commands = {c["name"] for c in manifest["provides"]["commands"]}
assert commands == EXPECTED_COMMANDS
def test_declares_no_hooks(self):
"""assess is a standalone pipeline: it must not register lifecycle
hooks (e.g. before_specify). Discovery and specification stay
separate processes; the only coupling is the forward decide ->
/speckit.specify handoff described in the commands."""
manifest = yaml.safe_load(
(EXT_DIR / "extension.yml").read_text(encoding="utf-8")
)
assert "hooks" not in manifest or not manifest["hooks"]
def test_readme_exists(self):
readme = EXT_DIR / "README.md"
assert readme.is_file()
text = readme.read_text(encoding="utf-8")
assert "Idea Assessment Pipeline Extension" in text
def test_command_files_exist(self):
for name in EXPECTED_COMMANDS:
cmd = EXT_DIR / "commands" / f"{name}.md"
assert cmd.is_file(), f"Missing command file: {cmd}"
# ── Catalog registration ─────────────────────────────────────────────────────
class TestCatalogEntry:
def test_catalog_lists_assess_as_bundled(self):
catalog = json.loads(
(PROJECT_ROOT / "extensions" / "catalog.json").read_text(encoding="utf-8")
)
entry = catalog["extensions"]["assess"]
assert entry["bundled"] is True
assert entry["id"] == "assess"
assert entry["author"] == "spec-kit-core"
# ── Bundle resolution ────────────────────────────────────────────────────────
class TestBundleResolution:
def test_locate_bundled_extension_finds_assess(self):
located = _locate_bundled_extension("assess")
assert located is not None
assert (located / "extension.yml").is_file()
# ── Install ──────────────────────────────────────────────────────────────────
class TestExtensionInstall:
def test_install_from_directory(self, tmp_path: Path):
from specify_cli.extensions import ExtensionManager
(tmp_path / ".specify").mkdir()
manager = ExtensionManager(tmp_path)
manifest = manager.install_from_directory(EXT_DIR, "0.9.0", register_commands=False)
assert manifest.id == "assess"
assert manager.registry.is_installed("assess")
installed = tmp_path / ".specify" / "extensions" / "assess"
for name in EXPECTED_COMMANDS:
assert (installed / "commands" / f"{name}.md").is_file()
def test_install_command_names(self, tmp_path: Path):
"""The installed manifest exposes the expected command names."""
from specify_cli.extensions import ExtensionManager
(tmp_path / ".specify").mkdir()
manager = ExtensionManager(tmp_path)
manifest = manager.install_from_directory(EXT_DIR, "0.9.0", register_commands=False)
names = {c["name"] for c in manifest.commands}
assert names == EXPECTED_COMMANDS