Files
github-spec-kit/tests/integrations/test_integration_copilot.py
Manfred Riem 53d9543355 feat: make agent-context extension a full opt-in (#3097)
* docs: add Spec Kit spec for agent-context full opt-in

Use Spec Kit's own specify workflow to author the spec that makes the
agent-context extension a full opt-in, removing all agent-context
configuration/support from the Python codebase and removing the
deprecation message. Force-added despite specs/ being gitignored; the
generated artifact will be purged prior to merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add Spec Kit plan artifacts for agent-context full opt-in

Phase 0/1 of the SDD plan workflow: plan.md, research.md, data-model.md,
quickstart.md, and contracts/cli-behavior.md. Constitution Check is a
documented no-op (repo has no ratified constitution). Force-added despite
specs/ being gitignored; generated artifacts will be purged prior to merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct Constitution Check against ratified v1.0.0

Earlier draft wrongly treated the gate as a no-op; the fork's main is 16
commits behind upstream/main, which carries .specify/memory/constitution.md.
Re-evaluate the feature against Principles I-V (all PASS) and note that
Principle I mandates keeping context_file as a declared class attribute,
validating the R1 metadata decision.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: refresh plan artifacts against synced upstream/main

After syncing fork main to upstream and rebasing, re-scan the current
agent-context surface. Upstream generalized the single context_file into a
plural context_files concept with new resolver helpers
(_resolve_context_files, _resolve_context_file_values,
_format_context_file_values) and upsert/remove now loop over multiple
files. Update research.md, data-model.md, contracts, quickstart grep
guards, and the plan summary to cover the expanded removal scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add Spec Kit tasks for agent-context full opt-in

Phase 2 of SDD: dependency-ordered tasks.md (30 tasks) organized by the
three user stories, with mandatory test tasks (Constitution Principle II)
and a foundational phase decoupling __CONTEXT_FILE__ resolution from the
extension config. Includes the extension self-seeding task (T015) and a
static guard test (T002) enforcing zero agent-context references in the CLI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat!: remove agent-context lifecycle from the Specify CLI

Make the agent-context extension a full opt-in. The CLI no longer
installs the extension during init, writes agent-context-config.yml,
or creates/updates/removes the managed Spec Kit section in agent
context files. Context-section upsert/remove, marker resolution,
extension-enabled gating, the config helpers, and the obsolete inline
deprecation warning are all removed. Integration context_file stays as
inert metadata; __CONTEXT_FILE__ now resolves from registry metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(agent-context): self-seed context file from the active integration

When agent-context-config.yml has no context_file/context_files, the
bundled bash and PowerShell update scripts now resolve the context file
from the active integration in .specify/init-options.json via the
integration registry, so the extension no longer depends on the CLI
writing its config.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test+docs: update suite and docs for agent-context opt-in

Update integration/extension tests to expect no agent-context install,
config, or context-section writes during init. Add a static guard test
(test_agent_context_cli_free.py) asserting the CLI source is free of
agent-context lifecycle symbols, plus backward-compatibility tests for
legacy projects. Refresh AGENTS.md, the extension README, and add a
CHANGELOG entry describing the opt-in behavior change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agent-context): warn on self-seed failure, correct docs, speed up guard test

Address PR review feedback:
- Self-seed scripts (bash + PowerShell) now emit an actionable warning when
  an active integration is configured but specify_cli cannot be imported by
  the chosen Python (e.g. pipx installs), or when the integration declares no
  context file, instead of silently falling through to 'nothing to do'.
- Correct the extension README disable note: command rendering never reads the
  extension config; __CONTEXT_FILE__ is always substituted from integration
  metadata, so a stale context_files value cannot affect rendering.
- Cache CLI source reads in the static guard test via a module-scoped fixture
  so the directory walk happens once instead of once per forbidden symbol.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(agent-context): ship self-owned per-agent context-file defaults

The extension now bundles agent-context-defaults.json (key→context_file
map) and self-seeds from it, dropping any dependency on the Specify CLI
registry. Both the bash and PowerShell update scripts read the bundled
JSON map keyed by the active integration from init-options.json.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat!: remove all agent-context state from the Specify CLI

Strip every context_file reference from the CLI: the field on all 35
integration classes, the IntegrationBase plumbing (process_template
param/step, _context_file_display, docstrings), the __CONTEXT_FILE__
resolution in agents.py, the legacy context_file/context_markers
popping in _helpers.py, and the context_file template in
integration_scaffold.py. Also drop the Agent context update step and
__CONTEXT_FILE__ placeholder from templates/commands/plan.md.

The agent-context extension now solely owns all context-file knowledge,
including the per-agent default mapping.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: drop context_file coverage and guard against CLI reintroduction

Remove CONTEXT_FILE attrs and context_file assertions across the base
mixins, all 35 per-integration test files, shared integration tests, and
conftest stubs. Rewrite the base-mixin context tests to assert no managed
section is written and no __CONTEXT_FILE__ placeholder survives. Extend
the CLI-free static guard to forbid context_file, __CONTEXT_FILE__, and
_context_file_display in src/specify_cli, and have the extension tests
copy the bundled defaults JSON so self-seed runs without the CLI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: reflect full removal of agent-context state from the CLI

Update AGENTS.md (integration examples, required-fields table, context
behavior section, pitfalls), CHANGELOG, and the SDD spec artifacts
(FR-007, SC-002, data-model) to state that the CLI carries no
context_file and the extension fully owns the per-agent default mapping
via agent-context-defaults.json.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: align SDD artifacts with full context_file removal

Update research.md (R1, R2, R4, summary table), contracts/cli-behavior.md
(C3, C5), tasks.md (Phase 2, T026, notes), plan.md (Principle I, source
map), and checklists/requirements.md so the spec artifacts reflect the
implemented decision: the CLI carries no context_file attribute or
__CONTEXT_FILE__ resolution, and the per-agent defaults map lives in the
extension. Resolves PR review #4548130110.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: scrub stale context-file mentions from CLI docstrings

Update the multi_install_safe docstring (drop the removed "context file"
invariant), the RovoDev setup docstring (no longer upserts a context
section), the Copilot module docstring (drop the context-file line), and
tighten the _update_init_options_for_integration note. Pure docstring
changes — no behavioral impact. Resolves PR review #4548237085.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test+docs: harden agent-context test helper and fix stale docs

- base.py: document multi_install_safe as an optional subclass attribute
  in the IntegrationBase docstring.
- test_cli.py: clarify the init-options assertion is guarding against
  leftover legacy agent-context keys, not relocation.
- test_extension_agent_context.py: _install_agent_context_config now
  asserts the bundled agent-context-defaults.json exists and always
  copies it, so self-seeding tests fail loudly instead of silently
  skipping when the map is missing.
- test_integration_cursor_agent.py: drop Path/IntegrationManifest imports
  left unused after removing the context-section frontmatter tests.

Resolves PR review #4548293116.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: remove gitignored SDD artifacts from specs/

The specs/001-agent-context-full-optin/ artifacts were force-added for
dogfooding visibility, but specs/ is gitignored and these were always
intended to be purged before merge. Remove them so merging does not add
an intentionally-untracked directory to repo history.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: keep CHANGELOG.md identical to upstream

CHANGELOG.md is auto-generated at release time, so the branch should not
carry a manual entry. Restore it to match upstream/main exactly.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: preserve Cursor .mdc frontmatter in agent-context updater scripts

The bundled agent-context updater scripts wrote the managed section as
plain text. For Cursor-style `.mdc` targets this dropped the required
`---\nalwaysApply: true\n---` frontmatter, reintroducing the rule-loading
bug originally fixed in #1699. Port the `_ensure_mdc_frontmatter` logic
into both the bash and PowerShell updaters: prepend frontmatter when
missing, repair `alwaysApply` when set to the wrong value, and leave
non-`.mdc` targets untouched. Add regression tests covering both shells.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: scope CLI-free guard to agent-context-specific symbols

Drop the bare "context_file" substring from FORBIDDEN_SYMBOLS so the
guard no longer fails on unrelated future CLI fields named context_file.
The list still covers agent-context-specific identifiers (__CONTEXT_FILE__,
_context_file_display, _resolve_context_files, _resolve_context_file_values).

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: harden agent-context bash self-seed against malformed init JSON

Two robustness fixes in the embedded Python self-seed logic:
- Coerce the integration value from init-options.json to a string only when
  it is actually a string; otherwise treat it as unset so a corrupted
  dict/list value degrades to the existing nothing-to-do behavior instead of
  breaking the agents-map lookup.
- Normalize agent-context-defaults.json: only use 'agents' when both the JSON
  root and the 'agents' value are dicts, so a wrong-shaped (but valid) JSON
  falls back to the warning path instead of raising on .get.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: correct PowerShell hyphenated key lookup and regex replace count

- Self-seed now reads the defaults mapping via
  $defaults.agents.PSObject.Properties[$integrationKey].Value instead of
  member access ($defaults.agents.$integrationKey), which parsed hyphenated
  keys like 'cursor-agent'/'kiro-cli' as subtraction and failed to resolve.
- Replace the static [regex]::Replace(..., 1) call, whose trailing 1 was
  interpreted as RegexOptions.IgnoreCase rather than a replacement count, with
  an instance Regex whose Replace(input, replacement, 1) limits to the first
  match as intended.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: make bash .mdc frontmatter guard case-insensitive

The bash updater only injected Cursor .mdc frontmatter when ctx_path ended
in lowercase '.mdc', so a mixed/upper-case extension (e.g. specify-rules.MDC)
was skipped and Cursor would not auto-load the rule file. Compare against the
casefolded path. The PowerShell variant already uses -match, which is
case-insensitive by default, so no change is needed there.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: document separator-agnostic agent-context update invocation

The README hard-coded the dot-notation slash command
(/speckit.agent-context.update), which hyphen-separator agents like Forge and
Cline do not recognize. Document the canonical command ID plus both slash
invocations so users copy the form their agent accepts.

Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-29 15:27:26 -05:00

756 lines
35 KiB
Python

"""Tests for CopilotIntegration."""
import json
import os
import yaml
from specify_cli.integrations import get_integration
from specify_cli.integrations.manifest import IntegrationManifest
class TestCopilotIntegration:
def test_copilot_key_and_config(self):
copilot = get_integration("copilot")
assert copilot is not None
assert copilot.key == "copilot"
assert copilot.config["folder"] == ".github/"
assert copilot.config["commands_subdir"] == "agents"
assert copilot.registrar_config["extension"] == ".agent.md"
def test_command_filename_agent_md(self):
copilot = get_integration("copilot")
assert copilot.command_filename("plan") == "speckit.plan.agent.md"
def test_setup_creates_agent_md_files(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
assert len(created) > 0
agent_files = [f for f in created if ".agent." in f.name]
assert len(agent_files) > 0
for f in agent_files:
assert f.parent == tmp_path / ".github" / "agents"
assert f.name.endswith(".agent.md")
def test_setup_creates_companion_prompts(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
prompt_files = [f for f in created if f.parent.name == "prompts"]
assert len(prompt_files) > 0
for f in prompt_files:
assert f.name.endswith(".prompt.md")
content = f.read_text(encoding="utf-8")
assert content.startswith("---\nagent: speckit.")
def test_agent_and_prompt_counts_match(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
agents = [f for f in created if ".agent.md" in f.name]
prompts = [f for f in created if ".prompt.md" in f.name]
assert len(agents) == len(prompts)
def test_setup_creates_vscode_settings_new(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
assert copilot._vscode_settings_path() is not None
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
settings = tmp_path / ".vscode" / "settings.json"
assert settings.exists()
assert settings in created
assert any("settings.json" in k for k in m.files)
def test_setup_merges_existing_vscode_settings(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
vscode_dir = tmp_path / ".vscode"
vscode_dir.mkdir(parents=True)
existing = {"editor.fontSize": 14, "custom.setting": True}
(vscode_dir / "settings.json").write_text(json.dumps(existing, indent=4), encoding="utf-8")
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
settings = tmp_path / ".vscode" / "settings.json"
data = json.loads(settings.read_text(encoding="utf-8"))
assert data["editor.fontSize"] == 14
assert data["custom.setting"] is True
assert settings not in created
assert not any("settings.json" in k for k in m.files)
def test_all_created_files_tracked_in_manifest(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m)
for f in created:
rel = f.resolve().relative_to(tmp_path.resolve()).as_posix()
assert rel in m.files, f"Created file {rel} not tracked in manifest"
def test_install_uninstall_roundtrip(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.install(tmp_path, m)
assert len(created) > 0
m.save()
for f in created:
assert f.exists()
removed, skipped = copilot.uninstall(tmp_path, m)
assert len(removed) == len(created)
assert skipped == []
def test_modified_file_survives_uninstall(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.install(tmp_path, m)
m.save()
modified_file = created[0]
modified_file.write_text("user modified this", encoding="utf-8")
removed, skipped = copilot.uninstall(tmp_path, m)
assert modified_file.exists()
assert modified_file in skipped
def test_directory_structure(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
copilot.setup(tmp_path, m)
agents_dir = tmp_path / ".github" / "agents"
assert agents_dir.is_dir()
agent_files = sorted(agents_dir.glob("speckit.*.agent.md"))
assert len(agent_files) == 10
expected_commands = {
"analyze", "clarify", "constitution", "converge", "implement",
"plan", "checklist", "specify", "tasks", "taskstoissues",
}
actual_commands = {f.name.removeprefix("speckit.").removesuffix(".agent.md") for f in agent_files}
assert actual_commands == expected_commands
def test_templates_are_processed(self, tmp_path):
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
copilot.setup(tmp_path, m)
agents_dir = tmp_path / ".github" / "agents"
for agent_file in agents_dir.glob("speckit.*.agent.md"):
content = agent_file.read_text(encoding="utf-8")
assert "{SCRIPT}" not in content, f"{agent_file.name} has unprocessed {{SCRIPT}}"
assert "__AGENT__" not in content, f"{agent_file.name} has unprocessed __AGENT__"
assert "{ARGS}" not in content, f"{agent_file.name} has unprocessed {{ARGS}}"
assert "__SPECKIT_COMMAND_" not in content, f"{agent_file.name} has unprocessed __SPECKIT_COMMAND_*__"
assert "\nscripts:\n" not in content
def test_specify_agent_resolves_active_spec_template(self, tmp_path):
"""Generated specify agent must not hardcode the core spec template."""
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
copilot.setup(tmp_path, m)
specify_file = tmp_path / ".github" / "agents" / "speckit.specify.agent.md"
content = specify_file.read_text(encoding="utf-8")
assert "specify preset resolve spec-template" in content
assert "resolved active `spec-template`" in content
assert "Copy `.specify/templates/spec-template.md`" not in content
assert "Load `.specify/templates/spec-template.md`" not in content
def test_plan_command_has_no_context_placeholder(self, tmp_path):
"""The core plan command must not carry a context-file placeholder —
agent context files are owned by the opt-in agent-context extension."""
from specify_cli.integrations.copilot import CopilotIntegration
copilot = CopilotIntegration()
m = IntegrationManifest("copilot", tmp_path)
copilot.setup(tmp_path, m)
plan_file = tmp_path / ".github" / "agents" / "speckit.plan.agent.md"
assert plan_file.exists()
content = plan_file.read_text(encoding="utf-8")
assert "__CONTEXT_FILE__" not in content
def test_complete_file_inventory_sh(self, tmp_path):
"""Every file produced by specify init --integration copilot --script sh."""
from typer.testing import CliRunner
from specify_cli import app
project = tmp_path / "inventory-sh"
project.mkdir()
old_cwd = os.getcwd()
try:
os.chdir(project)
result = CliRunner().invoke(app, [
"init", "--here", "--integration", "copilot", "--script", "sh",
], catch_exceptions=False)
finally:
os.chdir(old_cwd)
assert result.exit_code == 0
actual = sorted(p.relative_to(project).as_posix() for p in project.rglob("*") if p.is_file() and ".git" not in p.parts)
expected = sorted([
".github/agents/speckit.analyze.agent.md",
".github/agents/speckit.checklist.agent.md",
".github/agents/speckit.clarify.agent.md",
".github/agents/speckit.constitution.agent.md",
".github/agents/speckit.converge.agent.md",
".github/agents/speckit.implement.agent.md",
".github/agents/speckit.plan.agent.md",
".github/agents/speckit.specify.agent.md",
".github/agents/speckit.tasks.agent.md",
".github/agents/speckit.taskstoissues.agent.md",
".github/prompts/speckit.analyze.prompt.md",
".github/prompts/speckit.checklist.prompt.md",
".github/prompts/speckit.clarify.prompt.md",
".github/prompts/speckit.constitution.prompt.md",
".github/prompts/speckit.converge.prompt.md",
".github/prompts/speckit.implement.prompt.md",
".github/prompts/speckit.plan.prompt.md",
".github/prompts/speckit.specify.prompt.md",
".github/prompts/speckit.tasks.prompt.md",
".github/prompts/speckit.taskstoissues.prompt.md",
".vscode/settings.json",
".specify/integration.json",
".specify/init-options.json",
".specify/integrations/copilot.manifest.json",
".specify/integrations/speckit.manifest.json",
".specify/scripts/bash/check-prerequisites.sh",
".specify/scripts/bash/common.sh",
".specify/scripts/bash/create-new-feature.sh",
".specify/scripts/bash/setup-plan.sh",
".specify/scripts/bash/setup-tasks.sh",
".specify/templates/checklist-template.md",
".specify/templates/constitution-template.md",
".specify/templates/plan-template.md",
".specify/templates/spec-template.md",
".specify/templates/tasks-template.md",
".specify/memory/constitution.md",
".specify/workflows/speckit/workflow.yml",
".specify/workflows/workflow-registry.json",
])
assert actual == expected, (
f"Missing: {sorted(set(expected) - set(actual))}\n"
f"Extra: {sorted(set(actual) - set(expected))}"
)
def test_complete_file_inventory_ps(self, tmp_path):
"""Every file produced by specify init --integration copilot --script ps."""
from typer.testing import CliRunner
from specify_cli import app
project = tmp_path / "inventory-ps"
project.mkdir()
old_cwd = os.getcwd()
try:
os.chdir(project)
result = CliRunner().invoke(app, [
"init", "--here", "--integration", "copilot", "--script", "ps",
], catch_exceptions=False)
finally:
os.chdir(old_cwd)
assert result.exit_code == 0
actual = sorted(p.relative_to(project).as_posix() for p in project.rglob("*") if p.is_file() and ".git" not in p.parts)
expected = sorted([
".github/agents/speckit.analyze.agent.md",
".github/agents/speckit.checklist.agent.md",
".github/agents/speckit.clarify.agent.md",
".github/agents/speckit.constitution.agent.md",
".github/agents/speckit.converge.agent.md",
".github/agents/speckit.implement.agent.md",
".github/agents/speckit.plan.agent.md",
".github/agents/speckit.specify.agent.md",
".github/agents/speckit.tasks.agent.md",
".github/agents/speckit.taskstoissues.agent.md",
".github/prompts/speckit.analyze.prompt.md",
".github/prompts/speckit.checklist.prompt.md",
".github/prompts/speckit.clarify.prompt.md",
".github/prompts/speckit.constitution.prompt.md",
".github/prompts/speckit.converge.prompt.md",
".github/prompts/speckit.implement.prompt.md",
".github/prompts/speckit.plan.prompt.md",
".github/prompts/speckit.specify.prompt.md",
".github/prompts/speckit.tasks.prompt.md",
".github/prompts/speckit.taskstoissues.prompt.md",
".vscode/settings.json",
".specify/integration.json",
".specify/init-options.json",
".specify/integrations/copilot.manifest.json",
".specify/integrations/speckit.manifest.json",
".specify/scripts/powershell/check-prerequisites.ps1",
".specify/scripts/powershell/common.ps1",
".specify/scripts/powershell/create-new-feature.ps1",
".specify/scripts/powershell/setup-plan.ps1",
".specify/scripts/powershell/setup-tasks.ps1",
".specify/templates/checklist-template.md",
".specify/templates/constitution-template.md",
".specify/templates/plan-template.md",
".specify/templates/spec-template.md",
".specify/templates/tasks-template.md",
".specify/memory/constitution.md",
".specify/workflows/speckit/workflow.yml",
".specify/workflows/workflow-registry.json",
])
assert actual == expected, (
f"Missing: {sorted(set(expected) - set(actual))}\n"
f"Extra: {sorted(set(actual) - set(expected))}"
)
class TestCopilotSkillsMode:
"""Tests for Copilot integration in --skills mode."""
_SKILL_COMMANDS = [
"analyze", "clarify", "constitution", "converge", "implement",
"plan", "checklist", "specify", "tasks", "taskstoissues",
]
def _make_copilot(self):
from specify_cli.integrations.copilot import CopilotIntegration
return CopilotIntegration()
def _setup_skills(self, copilot, tmp_path):
m = IntegrationManifest("copilot", tmp_path)
created = copilot.setup(tmp_path, m, parsed_options={"skills": True})
return created, m
# -- Options ----------------------------------------------------------
def test_options_include_skills_flag(self):
copilot = get_integration("copilot")
opts = copilot.options()
skills_opts = [o for o in opts if o.name == "--skills"]
assert len(skills_opts) == 1
assert skills_opts[0].is_flag is True
assert skills_opts[0].default is False
# -- Skills directory structure ---------------------------------------
def test_skills_creates_skill_files(self, tmp_path):
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
assert len(created) > 0
skill_files = [f for f in created if f.name == "SKILL.md"]
assert len(skill_files) > 0
for f in skill_files:
assert f.exists()
assert f.parent.name.startswith("speckit-")
def test_skills_directory_under_github_skills(self, tmp_path):
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skills_dir = tmp_path / ".github" / "skills"
assert skills_dir.is_dir()
skill_files = [f for f in created if f.name == "SKILL.md"]
for f in skill_files:
assert f.resolve().parent.parent == skills_dir.resolve(), (
f"{f} is not under {skills_dir}"
)
def test_skills_directory_structure(self, tmp_path):
"""Each command produces speckit-<name>/SKILL.md."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
expected_commands = set(self._SKILL_COMMANDS)
actual_commands = set()
for f in skill_files:
skill_dir_name = f.parent.name
assert skill_dir_name.startswith("speckit-")
actual_commands.add(skill_dir_name.removeprefix("speckit-"))
assert actual_commands == expected_commands
# -- No companion files in skills mode --------------------------------
def test_skills_no_prompt_md_companions(self, tmp_path):
"""Skills mode must not generate .prompt.md companion files."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
prompt_files = [f for f in created if f.name.endswith(".prompt.md")]
assert prompt_files == []
prompts_dir = tmp_path / ".github" / "prompts"
if prompts_dir.exists():
assert list(prompts_dir.iterdir()) == []
def test_skills_no_vscode_settings(self, tmp_path):
"""Skills mode must not create or merge .vscode/settings.json."""
copilot = self._make_copilot()
self._setup_skills(copilot, tmp_path)
settings = tmp_path / ".vscode" / "settings.json"
assert not settings.exists()
def test_skills_no_agent_md_files(self, tmp_path):
"""Skills mode must not produce .agent.md files."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
agent_files = [f for f in created if f.name.endswith(".agent.md")]
assert agent_files == []
# -- Frontmatter structure --------------------------------------------
def test_skill_frontmatter_structure(self, tmp_path):
"""SKILL.md must have name, description, compatibility, metadata."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
for f in skill_files:
content = f.read_text(encoding="utf-8")
assert content.startswith("---\n"), f"{f} missing frontmatter"
parts = content.split("---", 2)
fm = yaml.safe_load(parts[1])
assert "name" in fm, f"{f} frontmatter missing 'name'"
assert "description" in fm, f"{f} frontmatter missing 'description'"
assert "compatibility" in fm, f"{f} frontmatter missing 'compatibility'"
assert "metadata" in fm, f"{f} frontmatter missing 'metadata'"
assert fm["metadata"]["author"] == "github-spec-kit"
# -- Copilot-specific post-processing ---------------------------------
def test_post_process_skill_content_does_not_inject_mode(self):
"""post_process_skill_content() must NOT inject mode: — VS Code Copilot does not support it."""
copilot = self._make_copilot()
content = (
"---\n"
'name: "speckit-plan"\n'
'description: "Plan workflow"\n'
"---\n"
"\nBody content\n"
)
updated = copilot.post_process_skill_content(content)
assert "mode:" not in updated
def test_post_process_skill_content_injects_hook_note(self):
"""post_process_skill_content() should inject shared hook guidance but not mode:."""
copilot = self._make_copilot()
content = (
"---\n"
'name: "speckit-specify"\n'
'description: "Specify workflow"\n'
"---\n"
"\n- For each executable hook, output the following\n"
)
updated = copilot.post_process_skill_content(content)
assert "replace dots" in updated
assert "mode:" not in updated
def test_post_process_idempotent(self):
"""post_process_skill_content() must be idempotent."""
copilot = self._make_copilot()
content = (
"---\n"
'name: "speckit-plan"\n'
'description: "Plan workflow"\n'
"---\n"
"\nBody content\n"
)
first = copilot.post_process_skill_content(content)
second = copilot.post_process_skill_content(first)
assert first == second
def test_skills_do_not_have_mode_in_frontmatter(self, tmp_path):
"""Generated SKILL.md files must NOT contain mode: — VS Code Copilot does not support it."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
assert len(skill_files) > 0
for f in skill_files:
content = f.read_text(encoding="utf-8")
parts = content.split("---", 2)
fm = yaml.safe_load(parts[1])
assert "mode" not in fm, f"{f} frontmatter must not contain unsupported 'mode' field"
def test_skills_hook_sections_explain_dotted_command_conversion(self, tmp_path):
"""Generated skills with hook sections should include shared hook guidance."""
copilot = self._make_copilot()
self._setup_skills(copilot, tmp_path)
specify_skill = tmp_path / ".github" / "skills" / "speckit-specify" / "SKILL.md"
content = specify_skill.read_text(encoding="utf-8")
assert "replace dots" in content
# -- Template processing ----------------------------------------------
def test_skills_templates_are_processed(self, tmp_path):
"""Skill body must have placeholders replaced."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
assert len(skill_files) > 0
for f in skill_files:
content = f.read_text(encoding="utf-8")
assert "{SCRIPT}" not in content, f"{f.name} has unprocessed {{SCRIPT}}"
assert "__AGENT__" not in content, f"{f.name} has unprocessed __AGENT__"
assert "{ARGS}" not in content, f"{f.name} has unprocessed {{ARGS}}"
assert "__SPECKIT_COMMAND_" not in content, f"{f.name} has unprocessed __SPECKIT_COMMAND_*__"
def test_skills_command_refs_use_hyphen(self, tmp_path):
"""Copilot skills mode must use /speckit-<name> not /speckit.<name>."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
assert len(skill_files) > 0
for f in skill_files:
content = f.read_text(encoding="utf-8")
assert "/speckit." not in content, (
f"{f.name} contains dot-notation /speckit. reference; "
f"skills mode must use /speckit-<name>"
)
def test_skills_mode_invoke_separator(self):
"""Copilot effective_invoke_separator should reflect skills mode."""
copilot = self._make_copilot()
assert copilot.effective_invoke_separator() == "."
assert copilot.effective_invoke_separator({"skills": True}) == "-"
assert copilot.effective_invoke_separator({"skills": False}) == "."
def test_skill_body_has_content(self, tmp_path):
"""Each SKILL.md body should contain template content."""
copilot = self._make_copilot()
created, _ = self._setup_skills(copilot, tmp_path)
skill_files = [f for f in created if f.name == "SKILL.md"]
for f in skill_files:
content = f.read_text(encoding="utf-8")
parts = content.split("---", 2)
body = parts[2].strip() if len(parts) >= 3 else ""
assert len(body) > 0, f"{f} has empty body"
def test_plan_skill_has_no_context_placeholder(self, tmp_path):
"""The core plan skill must not carry a context-file placeholder —
agent context files are owned by the opt-in agent-context extension."""
copilot = self._make_copilot()
self._setup_skills(copilot, tmp_path)
plan_file = tmp_path / ".github" / "skills" / "speckit-plan" / "SKILL.md"
assert plan_file.exists()
content = plan_file.read_text(encoding="utf-8")
assert "__CONTEXT_FILE__" not in content
# -- Manifest tracking ------------------------------------------------
def test_all_files_tracked_in_manifest(self, tmp_path):
copilot = self._make_copilot()
created, m = self._setup_skills(copilot, tmp_path)
for f in created:
rel = f.resolve().relative_to(tmp_path.resolve()).as_posix()
assert rel in m.files, f"{rel} not tracked in manifest"
# -- Install/uninstall roundtrip --------------------------------------
def test_install_uninstall_roundtrip(self, tmp_path):
copilot = self._make_copilot()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.install(tmp_path, m, parsed_options={"skills": True})
assert len(created) > 0
m.save()
for f in created:
assert f.exists()
removed, skipped = copilot.uninstall(tmp_path, m)
assert len(removed) == len(created)
assert skipped == []
def test_modified_file_survives_uninstall(self, tmp_path):
copilot = self._make_copilot()
m = IntegrationManifest("copilot", tmp_path)
created = copilot.install(tmp_path, m, parsed_options={"skills": True})
m.save()
modified_file = created[0]
modified_file.write_text("user modified this", encoding="utf-8")
removed, skipped = copilot.uninstall(tmp_path, m)
assert modified_file.exists()
assert modified_file in skipped
# -- build_command_invocation -----------------------------------------
def test_build_command_invocation_skills_mode(self):
copilot = self._make_copilot()
copilot._skills_mode = True
assert copilot.build_command_invocation("speckit.plan") == "/speckit-plan"
assert copilot.build_command_invocation("plan") == "/speckit-plan"
assert copilot.build_command_invocation("plan", "my args") == "/speckit-plan my args"
def test_build_command_invocation_skills_extension_command(self):
copilot = self._make_copilot()
copilot._skills_mode = True
assert copilot.build_command_invocation("speckit.git.commit") == "/speckit-git-commit"
assert copilot.build_command_invocation("git.commit") == "/speckit-git-commit"
def test_build_command_invocation_default_mode(self):
copilot = self._make_copilot()
assert copilot.build_command_invocation("plan", "my args") == "my args"
assert copilot.build_command_invocation("plan") == ""
# -- Context section ---------------------------------------------------
def test_skills_setup_does_not_write_context_section(self, tmp_path):
copilot = self._make_copilot()
self._setup_skills(copilot, tmp_path)
for path in tmp_path.rglob("*"):
if path.is_file():
text = path.read_text(encoding="utf-8", errors="ignore")
assert "<!-- SPECKIT START -->" not in text
# -- CLI integration test ---------------------------------------------
def test_init_with_integration_options_skills(self, tmp_path):
"""specify init --integration copilot --integration-options='--skills' scaffolds skills."""
from typer.testing import CliRunner
from specify_cli import app
project = tmp_path / "copilot-skills"
project.mkdir()
old_cwd = os.getcwd()
try:
os.chdir(project)
result = CliRunner().invoke(app, [
"init", "--here", "--integration", "copilot",
"--integration-options", "--skills",
"--script", "sh",
], catch_exceptions=False)
finally:
os.chdir(old_cwd)
assert result.exit_code == 0, f"init failed: {result.output}"
skills_dir = project / ".github" / "skills"
assert skills_dir.is_dir(), "Skills directory was not created"
plan_skill = skills_dir / "speckit-plan" / "SKILL.md"
assert plan_skill.exists(), "speckit-plan/SKILL.md not found"
# Verify no default-mode artifacts
assert not (project / ".github" / "agents").exists()
assert not (project / ".github" / "prompts").exists()
assert not (project / ".vscode" / "settings.json").exists()
def test_complete_file_inventory_skills_sh(self, tmp_path):
"""Every file produced by specify init --integration copilot --integration-options='--skills' --script sh."""
from typer.testing import CliRunner
from specify_cli import app
project = tmp_path / "inventory-skills-sh"
project.mkdir()
old_cwd = os.getcwd()
try:
os.chdir(project)
result = CliRunner().invoke(app, [
"init", "--here", "--integration", "copilot",
"--integration-options", "--skills",
"--script", "sh",
], catch_exceptions=False)
finally:
os.chdir(old_cwd)
assert result.exit_code == 0, f"init failed: {result.output}"
actual = sorted(p.relative_to(project).as_posix() for p in project.rglob("*") if p.is_file() and ".git" not in p.parts)
expected = sorted([
# Skill files (core commands)
*[f".github/skills/speckit-{cmd}/SKILL.md" for cmd in self._SKILL_COMMANDS],
# Integration metadata
".specify/init-options.json",
".specify/integration.json",
".specify/integrations/copilot.manifest.json",
".specify/integrations/speckit.manifest.json",
# Scripts (sh)
".specify/scripts/bash/check-prerequisites.sh",
".specify/scripts/bash/common.sh",
".specify/scripts/bash/create-new-feature.sh",
".specify/scripts/bash/setup-plan.sh",
".specify/scripts/bash/setup-tasks.sh",
# Templates
".specify/templates/checklist-template.md",
".specify/templates/constitution-template.md",
".specify/templates/plan-template.md",
".specify/templates/spec-template.md",
".specify/templates/tasks-template.md",
".specify/memory/constitution.md",
# Bundled workflow
".specify/workflows/speckit/workflow.yml",
".specify/workflows/workflow-registry.json",
])
assert actual == expected, (
f"Missing: {sorted(set(expected) - set(actual))}\n"
f"Extra: {sorted(set(actual) - set(expected))}"
)
# -- Singleton leak: _skills_mode must reset --------------------------
def test_skills_mode_resets_on_default_setup(self, tmp_path):
"""setup() with skills=True then without must reset _skills_mode."""
copilot = self._make_copilot()
# First call: skills mode
(tmp_path / "proj1").mkdir()
m1 = IntegrationManifest("copilot", tmp_path / "proj1")
copilot.setup(tmp_path / "proj1", m1, parsed_options={"skills": True})
assert copilot._skills_mode is True
# Second call: default mode (no skills option)
(tmp_path / "proj2").mkdir()
m2 = IntegrationManifest("copilot", tmp_path / "proj2")
copilot.setup(tmp_path / "proj2", m2)
assert copilot._skills_mode is False
# build_command_invocation must use default (dotted) mode
assert copilot.build_command_invocation("plan", "args") == "args"
# -- Auto-detection must ignore unrelated .github/skills/ -------------
def test_dispatch_ignores_unrelated_skills_directory(self, tmp_path):
"""dispatch_command() must not treat unrelated .github/skills/ as skills mode."""
copilot = self._make_copilot()
# Create a .github/skills/ with non-speckit content (e.g. GitHub Skills training)
unrelated = tmp_path / ".github" / "skills" / "introduction-to-github"
unrelated.mkdir(parents=True)
(unrelated / "README.md").write_text("# GitHub Skills training\n")
# Should NOT detect skills mode — cli_args should contain --agent
import unittest.mock as mock
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.Mock(returncode=0, stdout="", stderr="")
copilot.dispatch_command("plan", "my args", project_root=tmp_path, stream=False)
call_args = mock_run.call_args[0][0]
assert "--agent" in call_args, (
f"Expected --agent in cli_args but got: {call_args}"
)
assert "speckit.plan" in call_args
def test_dispatch_detects_speckit_skills_layout(self, tmp_path):
"""dispatch_command() detects speckit-*/SKILL.md as skills mode."""
copilot = self._make_copilot()
skill_dir = tmp_path / ".github" / "skills" / "speckit-plan"
skill_dir.mkdir(parents=True)
(skill_dir / "SKILL.md").write_text("---\nname: speckit-plan\n---\n")
import unittest.mock as mock
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.Mock(returncode=0, stdout="", stderr="")
copilot.dispatch_command("plan", "my args", project_root=tmp_path, stream=False)
call_args = mock_run.call_args[0][0]
assert "--agent" not in call_args, (
f"Skills mode should not use --agent, got: {call_args}"
)
prompt = call_args[call_args.index("-p") + 1]
assert "/speckit-plan" in prompt, (
f"Skills mode prompt should invoke /speckit-plan, got: {prompt}"
)
assert "my args" in prompt, (
f"Skills mode prompt should preserve user args, got: {prompt}"
)
# -- Next-steps display for Copilot skills mode -----------------------
def test_init_skills_next_steps_show_skill_syntax(self, tmp_path):
"""specify init --integration copilot --integration-options='--skills' shows /speckit-plan not /speckit.plan."""
from typer.testing import CliRunner
from specify_cli import app
project = tmp_path / "copilot-nextsteps"
project.mkdir()
old_cwd = os.getcwd()
try:
os.chdir(project)
result = CliRunner().invoke(app, [
"init", "--here", "--integration", "copilot",
"--integration-options", "--skills",
], catch_exceptions=False)
finally:
os.chdir(old_cwd)
assert result.exit_code == 0, f"init failed: {result.output}"
# Skills mode should show /speckit-plan (hyphenated)
assert "/speckit-plan" in result.output, (
f"Expected /speckit-plan in next steps but got:\n{result.output}"
)
# Must NOT show the dotted /speckit.plan form
assert "/speckit.plan" not in result.output, (
f"Should not show /speckit.plan in skills mode:\n{result.output}"
)