mirror of
https://github.com/github/spec-kit.git
synced 2026-07-06 14:01:01 +08:00
Add workflow engine with catalog system (#2158)
* Initial plan * Add workflow engine with step registry, expression engine, catalog system, and CLI commands Agent-Logs-Url: https://github.com/github/spec-kit/sessions/72a7bb5d-071f-4d67-a507-7e1abae2384d Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * Add comprehensive tests for workflow engine (94 tests) Agent-Logs-Url: https://github.com/github/spec-kit/sessions/72a7bb5d-071f-4d67-a507-7e1abae2384d Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * Address review feedback: do-while condition preservation and URL scheme validation Agent-Logs-Url: https://github.com/github/spec-kit/sessions/72a7bb5d-071f-4d67-a507-7e1abae2384d Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * Address review feedback, add CLI dispatch, interactive gates, and docs Review comments (7/7): - Add explanatory comment to empty except block - Implement workflow catalog download with cleanup on failure - Add input type coercion for number/boolean/enum - Fix example workflow to remove non-existent output references - Fix while_loop and if_then condition defaults (string 'false' → bool False) - Fix resume step index tracking with step_offset parameter CLI dispatch: - Add build_exec_args() and dispatch_command() to IntegrationBase - Override for Claude (skills: /speckit-specify), Gemini (-m flag), Codex (codex exec), Copilot (--agent speckit.specify) - CommandStep invokes installed commands by name via integration CLI - Add PromptStep for arbitrary inline prompts (10th step type) - Stream CLI output live to terminal (no silent blocking) - Remove timeout when streaming (user can Ctrl+C) - Ctrl+C saves state as PAUSED for clean resume Interactive gates: - Gate steps prompt [1] approve [2] reject in TTY - Fall back to PAUSED in non-interactive environments - Resume re-executes the gate for interactive prompting Documentation: - workflows/README.md — user guide - workflows/ARCHITECTURE.md — internals with Mermaid diagrams - workflows/PUBLISHING.md — catalog submission guide Tests: 94 → 122 workflow tests, 1362 total (all passing) * Fix ruff lint errors: unused imports, f-string placeholders, undefined name * Address second review: registry-backed validation, shell failures, loop/fan-out execution, URL validation - VALID_STEP_TYPES now queries STEP_REGISTRY dynamically - Shell step returns FAILED on non-zero exit code - Persist workflow YAML in run directory for reliable resume - Resume loads from run copy, falls back to installed workflow - Engine iterates while/do-while loops up to max_iterations - Engine expands fan-out per item with context.item - HTTPS URL validation for catalog workflow installs (HTTP allowed for localhost) - Fix catalog merge priority docstring (lower number wins) - Fix dispatch_command docstring (no build_exec_args_for_command) - Gate on_reject=retry pauses for re-prompt on resume - Update docs to 10 step types, add prompt step to tables and README * Potential fix for pull request finding 'Empty except' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Address third review: fan-out IDs, catalog guards, shell coercion, docs - Fan-out generates unique per-item step IDs and collects results - Catalog merge skips non-dict workflow entries (malformed data guard) - Shell step coerces run_cmd to str after expression evaluation - urlopen timeout=30 for catalog workflow installs - yaml.dump with sort_keys=False, allow_unicode=True for catalog configs - Document streaming timeout as intentionally unbounded (user Ctrl+C) - Document --allow-all-tools as required for non-interactive + future enhancement - Update test docstring and PUBLISHING.md to 10 step types with prompt * Validate final URL after redirects in catalog fetch urlopen follows redirects, so validate the response URL against the same HTTPS/localhost rules to prevent redirect-based downgrade attacks. * Address fourth review: filter arg eval, tags normalization, install redirect check - Filter arguments now evaluated via _evaluate_simple_expression() so default(42) returns int not string - Tags normalized: non-list/non-string values handled gracefully - Install URL redirect validation (same as catalog fetch) - Remove unused 'skipped' variable in catalog config parsing - Author 'github' → 'GitHub' in example workflow - Document nested step resume limitation (re-runs parent step) * Add explanatory comment to empty except ValueError block * Address fifth review: expression parsing, fan-out output, URL install, gate options - Move string literal parsing before operator detection in expressions so quoted strings with operators (e.g. 'a in b') are not mis-parsed - Fan-out: remove max_concurrency from persisted output, fix docstring to reflect sequential execution - workflow add: support URL sources with HTTPS/redirect validation, validate workflow ID is non-empty before writing files - Deduplicate local install logic via _validate_and_install_local() - Remove 'edit' gate option from speckit workflow (not implemented) * Add comments to empty except ValueError blocks in URL install * Address sixth review: operator precedence, fan_in cleanup, registry resilience, docs - Fix or/and operator precedence (or parsed first = lower precedence) - Restore context.fan_in after fan-in step completes - Catch JSONDecodeError in registry load for corrupted files - Replace print() with on_step_start callback (library-safe) - Gate validation warns when on_reject set but no reject option - Shell step: document shell=True security tradeoff - README: sdd-pipeline → speckit, parallel → sequential for fan-out - ARCHITECTURE.md: parallel → fan-out/fan-in in diagram * Address seventh review: string literal before pipe, type annotations, validate on install - Move string literal check above pipe filter parsing so 'a | b' works - Fix type annotations: input_values list[str] | None, run_id str | None - Run validate_workflow() before installing from local path/URL - Remove duplicate string literal check from expression parser * Address eighth review: fan-out namespaced IDs, early return, catalog validation - Fan-out per-item step IDs use _fanout_{step_id}_{base}_{idx} namespace to avoid collisions with user-defined step IDs - Early return after fan-out loop when state is paused/failed/aborted - Catalog installs parse + validate downloaded YAML before registering, using definition metadata instead of catalog entry for registry * Address ninth review: populate catalog, fix indentation, priority, README - Add speckit workflow entry to catalog.json so it's discoverable - Fix shell step output dict indentation - Catalog add_catalog priority derived from max existing + 1 - README Quick Start clarified with install + local file examples * Address tenth review: max_iterations validation, catalog config guard, version alignment - Validate max_iterations is int >= 1 in while and do-while steps - Guard add_catalog against corrupted config (non-dict/non-list) - Align speckit_version requirement to >=0.6.1 (current package version) - Fan-out template validation uses separate seen_ids set to avoid false duplication errors with user-defined step IDs * Address eleventh review: command step fails without CLI, ID mismatch warning, state persistence - Command step returns FAILED when CLI not installed (was silent COMPLETED) - Catalog install warns on workflow ID vs catalog key mismatch - Engine persists state.save() before returning on unknown step type - Update tests to expect FAILED for command steps without CLI - Integration tests use shell steps for CLI-independent execution * Address twelfth review: type annotations, version examples, streaming docs, requires - Fix workflow_search type annotations (str | None) - PUBLISHING.md: speckit_version >=0.15.0 → >=0.6.1 - Document that exit_code is captured and referenceable by later steps - Mark requires as declared-but-not-enforced (planned enhancement) - Note full stdout/stderr capture as planned enhancement * Enforce catalog key matches workflow ID (fail instead of warn) * Bundle speckit workflow: auto-install during specify init - Add workflows/speckit to pyproject.toml force-include for wheel builds - Add _locate_bundled_workflow() helper (mirrors _locate_bundled_extension) - Auto-install speckit workflow during specify init (after git extension) - Update all integration file inventory tests to expect workflow files * Address fourteenth review: prompt fails without CLI, resolved step data, fan-out normalization - PromptStep returns FAILED when CLI not installed (was silent COMPLETED) - Engine step_data prefers resolved values from step output - Fan-out normalizes output.results=[] for empty item lists - subprocess.run inherits stdout/stderr (no explicit sys.stdout) - Registry tests use issubset for extensibility * Address fifteenth review: fan_in docstring, gate defaults, validation guards, reserved prefix - FanInStep docstring: aggregate-only, no blocking semantics - FanInStep: guard output_config as dict, handle None - Gate validate: use same default options as execute - Validate inputs is dict and steps is list before iterating - Reserve _fanout_ prefix in step ID validation - PUBLISHING.md: remove unenforced checklist items, add _fanout_ note * Address sixteenth review: docs regex, fan_in try/finally, hyphenated dot-path keys - PUBLISHING.md: update ID regex docs to match implementation (single-char OK) - FanInStep: wrap expression evaluation in try/finally for context.fan_in - Expression dot-path: allow hyphens in keys before list index (e.g. run-tests[0]) * Make speckit workflow integration-agnostic, document Copilot CLI requirement - Workflow integration selectable via input (default: claude) - Each command step uses {{ inputs.integration }} instead of hardcoded copilot - Copilot docstring documents CLI requirement for workflow dispatch - Added install_url for Copilot CLI docs * Address seventeenth review: project checks, catalog robustness - Add .specify/ project check to workflow run/resume/status/search/info - remove_catalog validates config shape (dict + list) before indexing - _fetch_single_catalog validates response is a dict - _get_merged_workflows raises when all catalogs fail to fetch - add_catalog guards against non-dict catalog entries in config * Address eighteenth review: condition coercion, gate abort result, while default, cache guard, resume log - evaluate_condition treats plain 'false'/'true' strings as booleans - Gate abort returns StepResult(FAILED) instead of raising exception so step output is persisted in state for inspection - while_loop max_iterations optional (default 10), validation aligned - Catalog cache fallback catches invalid JSON gracefully - resume() appends workflow_finished log entry like execute() * Address nineteenth review: allow-all-tools opt-in, empty catalogs, abort dead code, while docstring - --allow-all-tools controlled by SPECKIT_ALLOW_ALL_TOOLS env var (default: 1) Set to 0 to disable automatic tool approval for Copilot CLI - Empty catalogs list falls back to built-in defaults (not an error) - Remove unreachable WorkflowAbortError catches from execute/resume (gate abort now returns StepResult(FAILED) instead of raising) - while_loop docstring updated: max_iterations is optional (default 10) * Address twentieth review: gate abort maps to ABORTED status, do-while max_iterations optional - Engine detects output.aborted from gate step and sets RunStatus.ABORTED (was unreachable — gate abort returned FAILED but status was always FAILED) - do-while max_iterations now optional (default 10), aligned with while_loop - do-while docstring and validation updated accordingly * Coerce default_options to dict, align bundled workflow ID regex with validator * Gate validates string options, prompt uses resolved integration, loop normalizes max_iterations * Use parentId:childId convention for nested step IDs - Fan-out per-item IDs use parentId:templateId:index (e.g. parallel:impl:0) - Reserve ':' in user step IDs (validation rejects) - Replaces _fanout_ prefix with cleaner namespacing - Expressions like {{ steps.parallel:impl:0.output.file }} work naturally * Validate workflow version is semantic versioning (X.Y.Z) * Schema version validation, strict semver, load_workflow docstring, preserve max_concurrency - Validate schema_version is '1.0' (reject unknown future schemas) - Strict semver regex: ^\d+\.\d+\.\d+$ (rejects 1.0.0beta etc.) - load_workflow docstring: 'parsed' not 'validated' - Keep max_concurrency in fan-out output (was dropped) - do_while docstring: engine re-evaluates step_config condition - ARCHITECTURE.md: document nested resume limitation * Path traversal prevention, loop step ID namespacing - RunState validates run_id is alphanumeric+hyphens (no path separators) - workflow_add validates catalog source doesn't escape workflows_dir - Loop iterations namespace nested step IDs as parentId:childId:iteration so multiple iterations don't overwrite each other in context/state --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This commit is contained in:
Copilot
68
src/specify_cli/workflows/__init__.py
Normal file
68
src/specify_cli/workflows/__init__.py
Normal file
@@ -0,0 +1,68 @@
|
||||
"""Workflow engine for multi-step, resumable automation workflows.
|
||||
|
||||
Provides:
|
||||
- ``StepBase`` — abstract base every step type must implement.
|
||||
- ``StepContext`` — execution context passed to each step.
|
||||
- ``StepResult`` — return value from step execution.
|
||||
- ``STEP_REGISTRY`` — maps ``type_key`` to ``StepBase`` subclass instances.
|
||||
- ``WorkflowEngine`` — orchestrator that loads, validates, and executes
|
||||
workflow YAML definitions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .base import StepBase
|
||||
|
||||
# Maps step type_key → StepBase instance.
|
||||
STEP_REGISTRY: dict[str, StepBase] = {}
|
||||
|
||||
|
||||
def _register_step(step: StepBase) -> None:
|
||||
"""Register a step type instance in the global registry.
|
||||
|
||||
Raises ``ValueError`` for falsy keys and ``KeyError`` for duplicates.
|
||||
"""
|
||||
key = step.type_key
|
||||
if not key:
|
||||
raise ValueError("Cannot register step type with an empty type_key.")
|
||||
if key in STEP_REGISTRY:
|
||||
raise KeyError(f"Step type with key {key!r} is already registered.")
|
||||
STEP_REGISTRY[key] = step
|
||||
|
||||
|
||||
def get_step_type(type_key: str) -> StepBase | None:
|
||||
"""Return the step type for *type_key*, or ``None`` if not registered."""
|
||||
return STEP_REGISTRY.get(type_key)
|
||||
|
||||
|
||||
# -- Register built-in step types ----------------------------------------
|
||||
|
||||
def _register_builtin_steps() -> None:
|
||||
"""Register all built-in step types."""
|
||||
from .steps.command import CommandStep
|
||||
from .steps.do_while import DoWhileStep
|
||||
from .steps.fan_in import FanInStep
|
||||
from .steps.fan_out import FanOutStep
|
||||
from .steps.gate import GateStep
|
||||
from .steps.if_then import IfThenStep
|
||||
from .steps.prompt import PromptStep
|
||||
from .steps.shell import ShellStep
|
||||
from .steps.switch import SwitchStep
|
||||
from .steps.while_loop import WhileStep
|
||||
|
||||
_register_step(CommandStep())
|
||||
_register_step(DoWhileStep())
|
||||
_register_step(FanInStep())
|
||||
_register_step(FanOutStep())
|
||||
_register_step(GateStep())
|
||||
_register_step(IfThenStep())
|
||||
_register_step(PromptStep())
|
||||
_register_step(ShellStep())
|
||||
_register_step(SwitchStep())
|
||||
_register_step(WhileStep())
|
||||
|
||||
|
||||
_register_builtin_steps()
|
||||
132
src/specify_cli/workflows/base.py
Normal file
132
src/specify_cli/workflows/base.py
Normal file
@@ -0,0 +1,132 @@
|
||||
"""Base classes for workflow step types.
|
||||
|
||||
Provides:
|
||||
- ``StepBase`` — abstract base every step type must implement.
|
||||
- ``StepContext`` — execution context passed to each step.
|
||||
- ``StepResult`` — return value from step execution.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
|
||||
class StepStatus(str, Enum):
|
||||
"""Status of a step execution."""
|
||||
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
SKIPPED = "skipped"
|
||||
PAUSED = "paused"
|
||||
|
||||
|
||||
class RunStatus(str, Enum):
|
||||
"""Status of a workflow run."""
|
||||
|
||||
CREATED = "created"
|
||||
RUNNING = "running"
|
||||
PAUSED = "paused"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
ABORTED = "aborted"
|
||||
|
||||
|
||||
@dataclass
|
||||
class StepContext:
|
||||
"""Execution context passed to each step.
|
||||
|
||||
Contains everything the step needs to resolve expressions, dispatch
|
||||
commands, and record results.
|
||||
"""
|
||||
|
||||
#: Resolved workflow inputs (from user prompts / defaults).
|
||||
inputs: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
#: Accumulated step results keyed by step ID.
|
||||
#: Each entry is ``{"integration": ..., "model": ..., "options": ...,
|
||||
#: "input": ..., "output": ...}``.
|
||||
steps: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
|
||||
#: Current fan-out item (set only inside fan-out iterations).
|
||||
item: Any = None
|
||||
|
||||
#: Fan-in aggregated results (set only for fan-in steps).
|
||||
fan_in: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
#: Workflow-level default integration key.
|
||||
default_integration: str | None = None
|
||||
|
||||
#: Workflow-level default model.
|
||||
default_model: str | None = None
|
||||
|
||||
#: Workflow-level default options.
|
||||
default_options: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
#: Project root path.
|
||||
project_root: str | None = None
|
||||
|
||||
#: Current run ID.
|
||||
run_id: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class StepResult:
|
||||
"""Return value from a step execution."""
|
||||
|
||||
#: Step status.
|
||||
status: StepStatus = StepStatus.COMPLETED
|
||||
|
||||
#: Output data (stored as ``steps.<id>.output``).
|
||||
output: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
#: Nested steps to execute (for control-flow steps like if/then).
|
||||
next_steps: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
#: Error message if step failed.
|
||||
error: str | None = None
|
||||
|
||||
|
||||
class StepBase(ABC):
|
||||
"""Abstract base class for workflow step types.
|
||||
|
||||
Every step type — built-in or extension-provided — implements this
|
||||
interface and registers in ``STEP_REGISTRY``.
|
||||
"""
|
||||
|
||||
#: Matches the ``type:`` value in workflow YAML.
|
||||
type_key: str = ""
|
||||
|
||||
@abstractmethod
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
"""Execute the step with the given config and context.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
config:
|
||||
The step configuration from workflow YAML.
|
||||
context:
|
||||
The execution context with inputs, accumulated step results, etc.
|
||||
|
||||
Returns
|
||||
-------
|
||||
StepResult with status, output data, and optional nested steps.
|
||||
"""
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
"""Validate step configuration and return a list of error messages.
|
||||
|
||||
An empty list means the configuration is valid.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
if "id" not in config:
|
||||
errors.append("Step is missing required 'id' field.")
|
||||
return errors
|
||||
|
||||
def can_resume(self, state: dict[str, Any]) -> bool:
|
||||
"""Return whether this step can be resumed from the given state."""
|
||||
return True
|
||||
540
src/specify_cli/workflows/catalog.py
Normal file
540
src/specify_cli/workflows/catalog.py
Normal file
@@ -0,0 +1,540 @@
|
||||
"""Workflow catalog — discovery, install, and management of workflows.
|
||||
|
||||
Mirrors the existing extension/preset catalog pattern with:
|
||||
- Multi-catalog stack (env var → project → user → built-in)
|
||||
- SHA256-hashed per-URL caching with 1-hour TTL
|
||||
- Workflow registry for installed workflow tracking
|
||||
- Search across all configured catalog sources
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Errors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class WorkflowCatalogError(Exception):
|
||||
"""Base error for workflow catalog operations."""
|
||||
|
||||
|
||||
class WorkflowValidationError(WorkflowCatalogError):
|
||||
"""Validation error for catalog config or workflow data."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CatalogEntry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkflowCatalogEntry:
|
||||
"""Represents a single catalog source in the catalog stack."""
|
||||
|
||||
url: str
|
||||
name: str
|
||||
priority: int
|
||||
install_allowed: bool
|
||||
description: str = ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WorkflowRegistry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class WorkflowRegistry:
|
||||
"""Manages the registry of installed workflows.
|
||||
|
||||
Tracks installed workflows and their metadata in
|
||||
``.specify/workflows/workflow-registry.json``.
|
||||
"""
|
||||
|
||||
REGISTRY_FILE = "workflow-registry.json"
|
||||
SCHEMA_VERSION = "1.0"
|
||||
|
||||
def __init__(self, project_root: Path) -> None:
|
||||
self.project_root = project_root
|
||||
self.workflows_dir = project_root / ".specify" / "workflows"
|
||||
self.registry_path = self.workflows_dir / self.REGISTRY_FILE
|
||||
self.data = self._load()
|
||||
|
||||
def _load(self) -> dict[str, Any]:
|
||||
"""Load registry from disk or create default."""
|
||||
if self.registry_path.exists():
|
||||
try:
|
||||
with open(self.registry_path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
# Corrupted registry file — reset to default
|
||||
return {"schema_version": self.SCHEMA_VERSION, "workflows": {}}
|
||||
return {"schema_version": self.SCHEMA_VERSION, "workflows": {}}
|
||||
|
||||
def save(self) -> None:
|
||||
"""Persist registry to disk."""
|
||||
self.workflows_dir.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.registry_path, "w", encoding="utf-8") as f:
|
||||
json.dump(self.data, f, indent=2)
|
||||
|
||||
def add(self, workflow_id: str, metadata: dict[str, Any]) -> None:
|
||||
"""Add or update an installed workflow entry."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
existing = self.data["workflows"].get(workflow_id, {})
|
||||
metadata["installed_at"] = existing.get(
|
||||
"installed_at", datetime.now(timezone.utc).isoformat()
|
||||
)
|
||||
metadata["updated_at"] = datetime.now(timezone.utc).isoformat()
|
||||
self.data["workflows"][workflow_id] = metadata
|
||||
self.save()
|
||||
|
||||
def remove(self, workflow_id: str) -> bool:
|
||||
"""Remove an installed workflow entry. Returns True if found."""
|
||||
if workflow_id in self.data["workflows"]:
|
||||
del self.data["workflows"][workflow_id]
|
||||
self.save()
|
||||
return True
|
||||
return False
|
||||
|
||||
def get(self, workflow_id: str) -> dict[str, Any] | None:
|
||||
"""Get metadata for an installed workflow."""
|
||||
return self.data["workflows"].get(workflow_id)
|
||||
|
||||
def list(self) -> dict[str, dict[str, Any]]:
|
||||
"""Return all installed workflows."""
|
||||
return dict(self.data["workflows"])
|
||||
|
||||
def is_installed(self, workflow_id: str) -> bool:
|
||||
"""Check if a workflow is installed."""
|
||||
return workflow_id in self.data["workflows"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WorkflowCatalog
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class WorkflowCatalog:
|
||||
"""Manages workflow catalog fetching, caching, and searching.
|
||||
|
||||
Resolution order for catalog sources:
|
||||
1. ``SPECKIT_WORKFLOW_CATALOG_URL`` env var (overrides all)
|
||||
2. Project-level ``.specify/workflow-catalogs.yml``
|
||||
3. User-level ``~/.specify/workflow-catalogs.yml``
|
||||
4. Built-in defaults (official + community)
|
||||
"""
|
||||
|
||||
DEFAULT_CATALOG_URL = (
|
||||
"https://raw.githubusercontent.com/github/spec-kit/main/"
|
||||
"workflows/catalog.json"
|
||||
)
|
||||
COMMUNITY_CATALOG_URL = (
|
||||
"https://raw.githubusercontent.com/github/spec-kit/main/"
|
||||
"workflows/catalog.community.json"
|
||||
)
|
||||
CACHE_DURATION = 3600 # 1 hour
|
||||
|
||||
def __init__(self, project_root: Path) -> None:
|
||||
self.project_root = project_root
|
||||
self.workflows_dir = project_root / ".specify" / "workflows"
|
||||
self.cache_dir = self.workflows_dir / ".cache"
|
||||
|
||||
# -- Catalog resolution -----------------------------------------------
|
||||
|
||||
def _validate_catalog_url(self, url: str) -> None:
|
||||
"""Validate that a catalog URL uses HTTPS (localhost HTTP allowed)."""
|
||||
from urllib.parse import urlparse
|
||||
|
||||
parsed = urlparse(url)
|
||||
is_localhost = parsed.hostname in ("localhost", "127.0.0.1", "::1")
|
||||
if parsed.scheme != "https" and not (
|
||||
parsed.scheme == "http" and is_localhost
|
||||
):
|
||||
raise WorkflowValidationError(
|
||||
f"Catalog URL must use HTTPS (got {parsed.scheme}://). "
|
||||
"HTTP is only allowed for localhost."
|
||||
)
|
||||
if not parsed.netloc:
|
||||
raise WorkflowValidationError(
|
||||
"Catalog URL must be a valid URL with a host."
|
||||
)
|
||||
|
||||
def _load_catalog_config(
|
||||
self, config_path: Path
|
||||
) -> list[WorkflowCatalogEntry] | None:
|
||||
"""Load catalog stack configuration from a YAML file."""
|
||||
if not config_path.exists():
|
||||
return None
|
||||
try:
|
||||
data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||
except (yaml.YAMLError, OSError, UnicodeError) as exc:
|
||||
raise WorkflowValidationError(
|
||||
f"Failed to read catalog config {config_path}: {exc}"
|
||||
)
|
||||
catalogs_data = data.get("catalogs", [])
|
||||
if not catalogs_data:
|
||||
# Empty catalogs list (e.g. after removing last entry)
|
||||
# is valid — fall back to built-in defaults.
|
||||
return None
|
||||
if not isinstance(catalogs_data, list):
|
||||
raise WorkflowValidationError(
|
||||
f"Invalid catalog config: 'catalogs' must be a list, "
|
||||
f"got {type(catalogs_data).__name__}"
|
||||
)
|
||||
|
||||
entries: list[WorkflowCatalogEntry] = []
|
||||
for idx, item in enumerate(catalogs_data):
|
||||
if not isinstance(item, dict):
|
||||
raise WorkflowValidationError(
|
||||
f"Invalid catalog entry at index {idx}: "
|
||||
f"expected a mapping, got {type(item).__name__}"
|
||||
)
|
||||
url = str(item.get("url", "")).strip()
|
||||
if not url:
|
||||
continue
|
||||
self._validate_catalog_url(url)
|
||||
try:
|
||||
priority = int(item.get("priority", idx + 1))
|
||||
except (TypeError, ValueError):
|
||||
raise WorkflowValidationError(
|
||||
f"Invalid priority for catalog "
|
||||
f"'{item.get('name', idx + 1)}': "
|
||||
f"expected integer, got {item.get('priority')!r}"
|
||||
)
|
||||
raw_install = item.get("install_allowed", False)
|
||||
if isinstance(raw_install, str):
|
||||
install_allowed = raw_install.strip().lower() in (
|
||||
"true",
|
||||
"yes",
|
||||
"1",
|
||||
)
|
||||
else:
|
||||
install_allowed = bool(raw_install)
|
||||
entries.append(
|
||||
WorkflowCatalogEntry(
|
||||
url=url,
|
||||
name=str(item.get("name", f"catalog-{idx + 1}")),
|
||||
priority=priority,
|
||||
install_allowed=install_allowed,
|
||||
description=str(item.get("description", "")),
|
||||
)
|
||||
)
|
||||
entries.sort(key=lambda e: e.priority)
|
||||
if not entries:
|
||||
raise WorkflowValidationError(
|
||||
f"Catalog config {config_path} contains {len(catalogs_data)} "
|
||||
f"entries but none have valid URLs."
|
||||
)
|
||||
return entries
|
||||
|
||||
def get_active_catalogs(self) -> list[WorkflowCatalogEntry]:
|
||||
"""Get the ordered list of active catalogs."""
|
||||
# 1. Environment variable override
|
||||
env_url = os.environ.get("SPECKIT_WORKFLOW_CATALOG_URL", "").strip()
|
||||
if env_url:
|
||||
self._validate_catalog_url(env_url)
|
||||
return [
|
||||
WorkflowCatalogEntry(
|
||||
url=env_url,
|
||||
name="env-override",
|
||||
priority=1,
|
||||
install_allowed=True,
|
||||
description="From SPECKIT_WORKFLOW_CATALOG_URL",
|
||||
)
|
||||
]
|
||||
|
||||
# 2. Project-level config
|
||||
project_config = self.project_root / ".specify" / "workflow-catalogs.yml"
|
||||
project_entries = self._load_catalog_config(project_config)
|
||||
if project_entries is not None:
|
||||
return project_entries
|
||||
|
||||
# 3. User-level config
|
||||
home = Path.home()
|
||||
user_config = home / ".specify" / "workflow-catalogs.yml"
|
||||
user_entries = self._load_catalog_config(user_config)
|
||||
if user_entries is not None:
|
||||
return user_entries
|
||||
|
||||
# 4. Built-in defaults
|
||||
return [
|
||||
WorkflowCatalogEntry(
|
||||
url=self.DEFAULT_CATALOG_URL,
|
||||
name="default",
|
||||
priority=1,
|
||||
install_allowed=True,
|
||||
description="Official workflows",
|
||||
),
|
||||
WorkflowCatalogEntry(
|
||||
url=self.COMMUNITY_CATALOG_URL,
|
||||
name="community",
|
||||
priority=2,
|
||||
install_allowed=False,
|
||||
description="Community-contributed workflows (discovery only)",
|
||||
),
|
||||
]
|
||||
|
||||
# -- Caching ----------------------------------------------------------
|
||||
|
||||
def _get_cache_paths(self, url: str) -> tuple[Path, Path]:
|
||||
"""Get cache file paths for a URL (hash-based)."""
|
||||
url_hash = hashlib.sha256(url.encode()).hexdigest()[:16]
|
||||
cache_file = self.cache_dir / f"workflow-catalog-{url_hash}.json"
|
||||
meta_file = self.cache_dir / f"workflow-catalog-{url_hash}-meta.json"
|
||||
return cache_file, meta_file
|
||||
|
||||
def _is_url_cache_valid(self, url: str) -> bool:
|
||||
"""Check if cached data for a URL is still fresh."""
|
||||
_, meta_file = self._get_cache_paths(url)
|
||||
if not meta_file.exists():
|
||||
return False
|
||||
try:
|
||||
with open(meta_file, encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
fetched_at = meta.get("fetched_at", 0)
|
||||
return (time.time() - fetched_at) < self.CACHE_DURATION
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return False
|
||||
|
||||
def _fetch_single_catalog(
|
||||
self, entry: WorkflowCatalogEntry, force_refresh: bool = False
|
||||
) -> dict[str, Any]:
|
||||
"""Fetch a single catalog, using cache when possible."""
|
||||
cache_file, meta_file = self._get_cache_paths(entry.url)
|
||||
|
||||
if not force_refresh and self._is_url_cache_valid(entry.url):
|
||||
try:
|
||||
with open(cache_file, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
|
||||
# Fetch from URL — validate scheme before opening and after redirects
|
||||
from urllib.parse import urlparse
|
||||
from urllib.request import urlopen
|
||||
|
||||
def _validate_catalog_url(url: str) -> None:
|
||||
parsed = urlparse(url)
|
||||
is_localhost = parsed.hostname in ("localhost", "127.0.0.1", "::1")
|
||||
if parsed.scheme != "https" and not (
|
||||
parsed.scheme == "http" and is_localhost
|
||||
):
|
||||
raise WorkflowCatalogError(
|
||||
f"Refusing to fetch catalog from non-HTTPS URL: {url}"
|
||||
)
|
||||
|
||||
_validate_catalog_url(entry.url)
|
||||
|
||||
try:
|
||||
with urlopen(entry.url, timeout=30) as resp: # noqa: S310
|
||||
_validate_catalog_url(resp.geturl())
|
||||
data = json.loads(resp.read().decode("utf-8"))
|
||||
except Exception as exc:
|
||||
# Fall back to cache if available
|
||||
if cache_file.exists():
|
||||
try:
|
||||
with open(cache_file, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, ValueError, OSError):
|
||||
pass
|
||||
raise WorkflowCatalogError(
|
||||
f"Failed to fetch catalog from {entry.url}: {exc}"
|
||||
) from exc
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise WorkflowCatalogError(
|
||||
f"Catalog from {entry.url} is not a valid JSON object."
|
||||
)
|
||||
|
||||
# Write cache
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
with open(cache_file, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
with open(meta_file, "w", encoding="utf-8") as f:
|
||||
json.dump({"url": entry.url, "fetched_at": time.time()}, f)
|
||||
|
||||
return data
|
||||
|
||||
def _get_merged_workflows(
|
||||
self, force_refresh: bool = False
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Merge workflows from all active catalogs (lower priority number wins)."""
|
||||
catalogs = self.get_active_catalogs()
|
||||
merged: dict[str, dict[str, Any]] = {}
|
||||
fetch_errors = 0
|
||||
|
||||
# Process later/higher-numbered entries first so earlier/lower-numbered
|
||||
# entries overwrite them on workflow ID conflicts.
|
||||
for entry in reversed(catalogs):
|
||||
try:
|
||||
data = self._fetch_single_catalog(entry, force_refresh)
|
||||
except WorkflowCatalogError:
|
||||
fetch_errors += 1
|
||||
continue
|
||||
workflows = data.get("workflows", {})
|
||||
# Handle both dict and list formats
|
||||
if isinstance(workflows, dict):
|
||||
for wf_id, wf_data in workflows.items():
|
||||
if not isinstance(wf_data, dict):
|
||||
continue
|
||||
wf_data["_catalog_name"] = entry.name
|
||||
wf_data["_install_allowed"] = entry.install_allowed
|
||||
merged[wf_id] = wf_data
|
||||
elif isinstance(workflows, list):
|
||||
for wf_data in workflows:
|
||||
if not isinstance(wf_data, dict):
|
||||
continue
|
||||
wf_id = wf_data.get("id", "")
|
||||
if wf_id:
|
||||
wf_data["_catalog_name"] = entry.name
|
||||
wf_data["_install_allowed"] = entry.install_allowed
|
||||
merged[wf_id] = wf_data
|
||||
if fetch_errors == len(catalogs) and catalogs:
|
||||
raise WorkflowCatalogError(
|
||||
"All configured catalogs failed to fetch."
|
||||
)
|
||||
return merged
|
||||
|
||||
# -- Public API -------------------------------------------------------
|
||||
|
||||
def search(
|
||||
self,
|
||||
query: str | None = None,
|
||||
tag: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Search workflows across all configured catalogs."""
|
||||
merged = self._get_merged_workflows()
|
||||
results: list[dict[str, Any]] = []
|
||||
|
||||
for wf_id, wf_data in merged.items():
|
||||
wf_data.setdefault("id", wf_id)
|
||||
if query:
|
||||
q = query.lower()
|
||||
searchable = " ".join(
|
||||
[
|
||||
wf_data.get("name", ""),
|
||||
wf_data.get("description", ""),
|
||||
wf_data.get("id", ""),
|
||||
]
|
||||
).lower()
|
||||
if q not in searchable:
|
||||
continue
|
||||
if tag:
|
||||
raw_tags = wf_data.get("tags", [])
|
||||
tags = raw_tags if isinstance(raw_tags, list) else []
|
||||
normalized_tags = [t.lower() for t in tags if isinstance(t, str)]
|
||||
if tag.lower() not in normalized_tags:
|
||||
continue
|
||||
results.append(wf_data)
|
||||
return results
|
||||
|
||||
def get_workflow_info(self, workflow_id: str) -> dict[str, Any] | None:
|
||||
"""Get details for a specific workflow from the catalog."""
|
||||
merged = self._get_merged_workflows()
|
||||
wf = merged.get(workflow_id)
|
||||
if wf:
|
||||
wf.setdefault("id", workflow_id)
|
||||
return wf
|
||||
|
||||
def get_catalog_configs(self) -> list[dict[str, Any]]:
|
||||
"""Return current catalog configuration as a list of dicts."""
|
||||
entries = self.get_active_catalogs()
|
||||
return [
|
||||
{
|
||||
"name": e.name,
|
||||
"url": e.url,
|
||||
"priority": e.priority,
|
||||
"install_allowed": e.install_allowed,
|
||||
"description": e.description,
|
||||
}
|
||||
for e in entries
|
||||
]
|
||||
|
||||
def add_catalog(self, url: str, name: str | None = None) -> None:
|
||||
"""Add a catalog source to the project-level config."""
|
||||
self._validate_catalog_url(url)
|
||||
config_path = self.project_root / ".specify" / "workflow-catalogs.yml"
|
||||
|
||||
data: dict[str, Any] = {"catalogs": []}
|
||||
if config_path.exists():
|
||||
raw = yaml.safe_load(config_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
raise WorkflowValidationError(
|
||||
"Catalog config file is corrupted (expected a mapping)."
|
||||
)
|
||||
data = raw
|
||||
|
||||
catalogs = data.get("catalogs", [])
|
||||
if not isinstance(catalogs, list):
|
||||
raise WorkflowValidationError(
|
||||
"Catalog config 'catalogs' must be a list."
|
||||
)
|
||||
# Check for duplicate URL (guard against non-dict entries)
|
||||
for cat in catalogs:
|
||||
if isinstance(cat, dict) and cat.get("url") == url:
|
||||
raise WorkflowValidationError(
|
||||
f"Catalog URL already configured: {url}"
|
||||
)
|
||||
|
||||
# Derive priority from the highest existing priority + 1
|
||||
max_priority = max(
|
||||
(cat.get("priority", 0) for cat in catalogs if isinstance(cat, dict)),
|
||||
default=0,
|
||||
)
|
||||
catalogs.append(
|
||||
{
|
||||
"name": name or f"catalog-{len(catalogs) + 1}",
|
||||
"url": url,
|
||||
"priority": max_priority + 1,
|
||||
"install_allowed": True,
|
||||
"description": "",
|
||||
}
|
||||
)
|
||||
data["catalogs"] = catalogs
|
||||
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)
|
||||
|
||||
def remove_catalog(self, index: int) -> str:
|
||||
"""Remove a catalog source by index (0-based). Returns the removed name."""
|
||||
config_path = self.project_root / ".specify" / "workflow-catalogs.yml"
|
||||
if not config_path.exists():
|
||||
raise WorkflowValidationError("No catalog config file found.")
|
||||
|
||||
data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||
if not isinstance(data, dict):
|
||||
raise WorkflowValidationError(
|
||||
"Catalog config file is corrupted (expected a mapping)."
|
||||
)
|
||||
catalogs = data.get("catalogs", [])
|
||||
if not isinstance(catalogs, list):
|
||||
raise WorkflowValidationError(
|
||||
"Catalog config 'catalogs' must be a list."
|
||||
)
|
||||
|
||||
if index < 0 or index >= len(catalogs):
|
||||
raise WorkflowValidationError(
|
||||
f"Catalog index {index} out of range (0-{len(catalogs) - 1})."
|
||||
)
|
||||
|
||||
removed = catalogs.pop(index)
|
||||
data["catalogs"] = catalogs
|
||||
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)
|
||||
|
||||
if isinstance(removed, dict):
|
||||
return removed.get("name", f"catalog-{index + 1}")
|
||||
return f"catalog-{index + 1}"
|
||||
778
src/specify_cli/workflows/engine.py
Normal file
778
src/specify_cli/workflows/engine.py
Normal file
@@ -0,0 +1,778 @@
|
||||
"""Workflow engine — loads, validates, and executes workflow YAML definitions.
|
||||
|
||||
The engine is the orchestrator that:
|
||||
- Parses workflow YAML definitions
|
||||
- Validates step configurations and requirements
|
||||
- Executes steps sequentially, dispatching to the correct step type
|
||||
- Manages state persistence for resume capability
|
||||
- Handles control flow (branching, loops, fan-out/fan-in)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from .base import RunStatus, StepContext, StepResult, StepStatus
|
||||
|
||||
|
||||
# -- Workflow Definition --------------------------------------------------
|
||||
|
||||
|
||||
class WorkflowDefinition:
|
||||
"""Parsed and validated workflow YAML definition."""
|
||||
|
||||
def __init__(self, data: dict[str, Any], source_path: Path | None = None) -> None:
|
||||
self.data = data
|
||||
self.source_path = source_path
|
||||
|
||||
workflow = data.get("workflow", {})
|
||||
self.id: str = workflow.get("id", "")
|
||||
self.name: str = workflow.get("name", "")
|
||||
self.version: str = workflow.get("version", "0.0.0")
|
||||
self.author: str = workflow.get("author", "")
|
||||
self.description: str = workflow.get("description", "")
|
||||
self.schema_version: str = data.get("schema_version", "1.0")
|
||||
|
||||
# Defaults
|
||||
self.default_integration: str | None = workflow.get("integration")
|
||||
self.default_model: str | None = workflow.get("model")
|
||||
self.default_options: dict[str, Any] = workflow.get("options") or {}
|
||||
if not isinstance(self.default_options, dict):
|
||||
self.default_options = {}
|
||||
|
||||
# Requirements (declared but not yet enforced at runtime;
|
||||
# enforcement is a planned enhancement)
|
||||
self.requires: dict[str, Any] = data.get("requires", {})
|
||||
|
||||
# Inputs
|
||||
self.inputs: dict[str, Any] = data.get("inputs", {})
|
||||
|
||||
# Steps
|
||||
self.steps: list[dict[str, Any]] = data.get("steps", [])
|
||||
|
||||
@classmethod
|
||||
def from_yaml(cls, path: Path) -> WorkflowDefinition:
|
||||
"""Load a workflow definition from a YAML file."""
|
||||
with open(path, encoding="utf-8") as f:
|
||||
data = yaml.safe_load(f)
|
||||
if not isinstance(data, dict):
|
||||
msg = f"Workflow YAML must be a mapping, got {type(data).__name__}."
|
||||
raise ValueError(msg)
|
||||
return cls(data, source_path=path)
|
||||
|
||||
@classmethod
|
||||
def from_string(cls, content: str) -> WorkflowDefinition:
|
||||
"""Load a workflow definition from a YAML string."""
|
||||
data = yaml.safe_load(content)
|
||||
if not isinstance(data, dict):
|
||||
msg = f"Workflow YAML must be a mapping, got {type(data).__name__}."
|
||||
raise ValueError(msg)
|
||||
return cls(data)
|
||||
|
||||
|
||||
# -- Workflow Validation --------------------------------------------------
|
||||
|
||||
# ID format: lowercase alphanumeric with hyphens
|
||||
_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$")
|
||||
|
||||
# Valid step types (matching STEP_REGISTRY keys)
|
||||
def _get_valid_step_types() -> set[str]:
|
||||
"""Return valid step types from the registry, with a built-in fallback."""
|
||||
from . import STEP_REGISTRY
|
||||
if STEP_REGISTRY:
|
||||
return set(STEP_REGISTRY.keys())
|
||||
return {
|
||||
"command", "shell", "prompt", "gate", "if",
|
||||
"switch", "while", "do-while", "fan-out", "fan-in",
|
||||
}
|
||||
|
||||
|
||||
def validate_workflow(definition: WorkflowDefinition) -> list[str]:
|
||||
"""Validate a workflow definition and return a list of error messages.
|
||||
|
||||
An empty list means the workflow is valid.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
|
||||
# -- Schema version ---------------------------------------------------
|
||||
if definition.schema_version not in ("1.0", "1"):
|
||||
errors.append(
|
||||
f"Unsupported schema_version {definition.schema_version!r}. "
|
||||
f"Expected '1.0'."
|
||||
)
|
||||
|
||||
# -- Top-level fields -------------------------------------------------
|
||||
if not definition.id:
|
||||
errors.append("Workflow is missing 'workflow.id'.")
|
||||
elif not _ID_PATTERN.match(definition.id):
|
||||
errors.append(
|
||||
f"Workflow ID {definition.id!r} must be lowercase alphanumeric "
|
||||
f"with hyphens."
|
||||
)
|
||||
|
||||
if not definition.name:
|
||||
errors.append("Workflow is missing 'workflow.name'.")
|
||||
|
||||
if not definition.version:
|
||||
errors.append("Workflow is missing 'workflow.version'.")
|
||||
elif not re.match(r"^\d+\.\d+\.\d+$", definition.version):
|
||||
errors.append(
|
||||
f"Workflow version {definition.version!r} is not valid "
|
||||
f"semantic versioning (expected X.Y.Z)."
|
||||
)
|
||||
|
||||
# -- Inputs -----------------------------------------------------------
|
||||
if not isinstance(definition.inputs, dict):
|
||||
errors.append("'inputs' must be a mapping (or omitted).")
|
||||
else:
|
||||
for input_name, input_def in definition.inputs.items():
|
||||
if not isinstance(input_def, dict):
|
||||
errors.append(f"Input {input_name!r} must be a mapping.")
|
||||
continue
|
||||
input_type = input_def.get("type")
|
||||
if input_type and input_type not in ("string", "number", "boolean"):
|
||||
errors.append(
|
||||
f"Input {input_name!r} has invalid type {input_type!r}. "
|
||||
f"Must be 'string', 'number', or 'boolean'."
|
||||
)
|
||||
|
||||
# -- Steps ------------------------------------------------------------
|
||||
if not isinstance(definition.steps, list):
|
||||
errors.append("'steps' must be a list.")
|
||||
return errors
|
||||
if not definition.steps:
|
||||
errors.append("Workflow has no steps defined.")
|
||||
|
||||
seen_ids: set[str] = set()
|
||||
_validate_steps(definition.steps, seen_ids, errors)
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def _validate_steps(
|
||||
steps: list[dict[str, Any]],
|
||||
seen_ids: set[str],
|
||||
errors: list[str],
|
||||
) -> None:
|
||||
"""Recursively validate a list of steps."""
|
||||
from . import STEP_REGISTRY
|
||||
|
||||
for step_config in steps:
|
||||
if not isinstance(step_config, dict):
|
||||
errors.append(f"Step must be a mapping, got {type(step_config).__name__}.")
|
||||
continue
|
||||
|
||||
step_id = step_config.get("id")
|
||||
if not step_id:
|
||||
errors.append("Step is missing 'id' field.")
|
||||
continue
|
||||
|
||||
if ":" in step_id:
|
||||
errors.append(
|
||||
f"Step ID {step_id!r} contains ':' which is reserved "
|
||||
f"for engine-generated nested IDs (parentId:childId)."
|
||||
)
|
||||
|
||||
if step_id in seen_ids:
|
||||
errors.append(f"Duplicate step ID {step_id!r}.")
|
||||
seen_ids.add(step_id)
|
||||
|
||||
# Determine step type
|
||||
step_type = step_config.get("type", "command")
|
||||
if step_type not in _get_valid_step_types():
|
||||
errors.append(
|
||||
f"Step {step_id!r} has invalid type {step_type!r}."
|
||||
)
|
||||
continue
|
||||
|
||||
# Delegate to step-specific validation
|
||||
step_impl = STEP_REGISTRY.get(step_type)
|
||||
if step_impl:
|
||||
step_errors = step_impl.validate(step_config)
|
||||
errors.extend(step_errors)
|
||||
|
||||
# Recursively validate nested steps
|
||||
for nested_key in ("then", "else", "steps"):
|
||||
nested = step_config.get(nested_key)
|
||||
if isinstance(nested, list):
|
||||
_validate_steps(nested, seen_ids, errors)
|
||||
|
||||
# Validate switch cases
|
||||
cases = step_config.get("cases")
|
||||
if isinstance(cases, dict):
|
||||
for _case_key, case_steps in cases.items():
|
||||
if isinstance(case_steps, list):
|
||||
_validate_steps(case_steps, seen_ids, errors)
|
||||
|
||||
# Validate switch default
|
||||
default = step_config.get("default")
|
||||
if isinstance(default, list):
|
||||
_validate_steps(default, seen_ids, errors)
|
||||
|
||||
# Validate fan-out nested step (template — not added to seen_ids
|
||||
# since the engine generates parentId:templateId:index at runtime)
|
||||
fan_step = step_config.get("step")
|
||||
if isinstance(fan_step, dict):
|
||||
fan_errors: list[str] = []
|
||||
_validate_steps([fan_step], set(), fan_errors)
|
||||
errors.extend(fan_errors)
|
||||
|
||||
|
||||
# -- Run State Persistence ------------------------------------------------
|
||||
|
||||
|
||||
class RunState:
|
||||
"""Manages workflow run state for persistence and resume."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
run_id: str | None = None,
|
||||
workflow_id: str = "",
|
||||
project_root: Path | None = None,
|
||||
) -> None:
|
||||
self.run_id = run_id or str(uuid.uuid4())[:8]
|
||||
if not re.match(r'^[a-zA-Z0-9][a-zA-Z0-9_-]*$', self.run_id):
|
||||
msg = f"Invalid run_id {self.run_id!r}: must be alphanumeric with hyphens/underscores only."
|
||||
raise ValueError(msg)
|
||||
self.workflow_id = workflow_id
|
||||
self.project_root = project_root or Path(".")
|
||||
self.status = RunStatus.CREATED
|
||||
self.current_step_index = 0
|
||||
self.current_step_id: str | None = None
|
||||
self.step_results: dict[str, dict[str, Any]] = {}
|
||||
self.inputs: dict[str, Any] = {}
|
||||
self.created_at = datetime.now(timezone.utc).isoformat()
|
||||
self.updated_at = self.created_at
|
||||
self.log_entries: list[dict[str, Any]] = []
|
||||
|
||||
@property
|
||||
def runs_dir(self) -> Path:
|
||||
return self.project_root / ".specify" / "workflows" / "runs" / self.run_id
|
||||
|
||||
def save(self) -> None:
|
||||
"""Persist current state to disk."""
|
||||
self.updated_at = datetime.now(timezone.utc).isoformat()
|
||||
runs_dir = self.runs_dir
|
||||
runs_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
state_data = {
|
||||
"run_id": self.run_id,
|
||||
"workflow_id": self.workflow_id,
|
||||
"status": self.status.value,
|
||||
"current_step_index": self.current_step_index,
|
||||
"current_step_id": self.current_step_id,
|
||||
"step_results": self.step_results,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
}
|
||||
with open(runs_dir / "state.json", "w", encoding="utf-8") as f:
|
||||
json.dump(state_data, f, indent=2)
|
||||
|
||||
inputs_data = {"inputs": self.inputs}
|
||||
with open(runs_dir / "inputs.json", "w", encoding="utf-8") as f:
|
||||
json.dump(inputs_data, f, indent=2)
|
||||
|
||||
@classmethod
|
||||
def load(cls, run_id: str, project_root: Path) -> RunState:
|
||||
"""Load a run state from disk."""
|
||||
runs_dir = project_root / ".specify" / "workflows" / "runs" / run_id
|
||||
state_path = runs_dir / "state.json"
|
||||
if not state_path.exists():
|
||||
msg = f"Run state not found: {state_path}"
|
||||
raise FileNotFoundError(msg)
|
||||
|
||||
with open(state_path, encoding="utf-8") as f:
|
||||
state_data = json.load(f)
|
||||
|
||||
state = cls(
|
||||
run_id=state_data["run_id"],
|
||||
workflow_id=state_data["workflow_id"],
|
||||
project_root=project_root,
|
||||
)
|
||||
state.status = RunStatus(state_data["status"])
|
||||
state.current_step_index = state_data.get("current_step_index", 0)
|
||||
state.current_step_id = state_data.get("current_step_id")
|
||||
state.step_results = state_data.get("step_results", {})
|
||||
state.created_at = state_data.get("created_at", "")
|
||||
state.updated_at = state_data.get("updated_at", "")
|
||||
|
||||
inputs_path = runs_dir / "inputs.json"
|
||||
if inputs_path.exists():
|
||||
with open(inputs_path, encoding="utf-8") as f:
|
||||
inputs_data = json.load(f)
|
||||
state.inputs = inputs_data.get("inputs", {})
|
||||
|
||||
return state
|
||||
|
||||
def append_log(self, entry: dict[str, Any]) -> None:
|
||||
"""Append a log entry to the run log."""
|
||||
entry["timestamp"] = datetime.now(timezone.utc).isoformat()
|
||||
self.log_entries.append(entry)
|
||||
|
||||
runs_dir = self.runs_dir
|
||||
runs_dir.mkdir(parents=True, exist_ok=True)
|
||||
with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
|
||||
|
||||
# -- Workflow Engine ------------------------------------------------------
|
||||
|
||||
|
||||
class WorkflowEngine:
|
||||
"""Orchestrator that loads, validates, and executes workflow definitions."""
|
||||
|
||||
def __init__(self, project_root: Path | None = None) -> None:
|
||||
self.project_root = project_root or Path(".")
|
||||
self.on_step_start: Any = None # Callable[[str, str], None] | None
|
||||
|
||||
def load_workflow(self, source: str | Path) -> WorkflowDefinition:
|
||||
"""Load a workflow from an installed ID or a local YAML path.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
source:
|
||||
Either a workflow ID (looked up in the installed workflows
|
||||
directory) or a path to a YAML file.
|
||||
|
||||
Returns
|
||||
-------
|
||||
A parsed ``WorkflowDefinition`` (not yet validated; call
|
||||
``validate_workflow()`` or ``engine.validate()`` separately).
|
||||
|
||||
Raises
|
||||
------
|
||||
FileNotFoundError:
|
||||
If the workflow file cannot be found.
|
||||
ValueError:
|
||||
If the workflow YAML is invalid.
|
||||
"""
|
||||
path = Path(source)
|
||||
|
||||
# Try as a direct file path first
|
||||
if path.suffix in (".yml", ".yaml") and path.exists():
|
||||
return WorkflowDefinition.from_yaml(path)
|
||||
|
||||
# Try as an installed workflow ID
|
||||
installed_path = (
|
||||
self.project_root
|
||||
/ ".specify"
|
||||
/ "workflows"
|
||||
/ str(source)
|
||||
/ "workflow.yml"
|
||||
)
|
||||
if installed_path.exists():
|
||||
return WorkflowDefinition.from_yaml(installed_path)
|
||||
|
||||
msg = f"Workflow not found: {source}"
|
||||
raise FileNotFoundError(msg)
|
||||
|
||||
def validate(self, definition: WorkflowDefinition) -> list[str]:
|
||||
"""Validate a workflow definition."""
|
||||
return validate_workflow(definition)
|
||||
|
||||
def execute(
|
||||
self,
|
||||
definition: WorkflowDefinition,
|
||||
inputs: dict[str, Any] | None = None,
|
||||
run_id: str | None = None,
|
||||
) -> RunState:
|
||||
"""Execute a workflow definition.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
definition:
|
||||
The validated workflow definition.
|
||||
inputs:
|
||||
User-provided input values.
|
||||
run_id:
|
||||
Optional run ID (auto-generated if not provided).
|
||||
|
||||
Returns
|
||||
-------
|
||||
The final ``RunState`` after execution completes (or pauses).
|
||||
"""
|
||||
from . import STEP_REGISTRY
|
||||
|
||||
state = RunState(
|
||||
run_id=run_id,
|
||||
workflow_id=definition.id,
|
||||
project_root=self.project_root,
|
||||
)
|
||||
|
||||
# Persist a copy of the workflow definition so resume can
|
||||
# reload it even if the original source is no longer available
|
||||
# (e.g. a local YAML path that was moved or deleted).
|
||||
run_dir = self.project_root / ".specify" / "workflows" / "runs" / state.run_id
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
workflow_copy = run_dir / "workflow.yml"
|
||||
import yaml
|
||||
with open(workflow_copy, "w", encoding="utf-8") as f:
|
||||
yaml.safe_dump(definition.data, f, sort_keys=False)
|
||||
|
||||
# Resolve inputs
|
||||
resolved_inputs = self._resolve_inputs(definition, inputs or {})
|
||||
state.inputs = resolved_inputs
|
||||
state.status = RunStatus.RUNNING
|
||||
state.save()
|
||||
|
||||
context = StepContext(
|
||||
inputs=resolved_inputs,
|
||||
default_integration=definition.default_integration,
|
||||
default_model=definition.default_model,
|
||||
default_options=definition.default_options,
|
||||
project_root=str(self.project_root),
|
||||
run_id=state.run_id,
|
||||
)
|
||||
|
||||
# Execute steps
|
||||
try:
|
||||
self._execute_steps(definition.steps, context, state, STEP_REGISTRY)
|
||||
except KeyboardInterrupt:
|
||||
state.status = RunStatus.PAUSED
|
||||
state.append_log({"event": "workflow_interrupted"})
|
||||
state.save()
|
||||
return state
|
||||
except Exception as exc:
|
||||
state.status = RunStatus.FAILED
|
||||
state.append_log({"event": "workflow_failed", "error": str(exc)})
|
||||
state.save()
|
||||
raise
|
||||
|
||||
if state.status == RunStatus.RUNNING:
|
||||
state.status = RunStatus.COMPLETED
|
||||
state.append_log({"event": "workflow_finished", "status": state.status.value})
|
||||
state.save()
|
||||
return state
|
||||
|
||||
def resume(self, run_id: str) -> RunState:
|
||||
"""Resume a paused or failed workflow run."""
|
||||
state = RunState.load(run_id, self.project_root)
|
||||
if state.status not in (RunStatus.PAUSED, RunStatus.FAILED):
|
||||
msg = f"Cannot resume run {run_id!r} with status {state.status.value!r}."
|
||||
raise ValueError(msg)
|
||||
|
||||
# Load the workflow definition — try the persisted copy in the
|
||||
# run directory first so resume works even if the original
|
||||
# source (e.g. a local YAML path) is no longer available.
|
||||
run_dir = self.project_root / ".specify" / "workflows" / "runs" / run_id
|
||||
run_copy = run_dir / "workflow.yml"
|
||||
if run_copy.exists():
|
||||
definition = WorkflowDefinition.from_yaml(run_copy)
|
||||
else:
|
||||
definition = self.load_workflow(state.workflow_id)
|
||||
|
||||
# Restore context
|
||||
context = StepContext(
|
||||
inputs=state.inputs,
|
||||
steps=state.step_results,
|
||||
default_integration=definition.default_integration,
|
||||
default_model=definition.default_model,
|
||||
default_options=definition.default_options,
|
||||
project_root=str(self.project_root),
|
||||
run_id=state.run_id,
|
||||
)
|
||||
|
||||
from . import STEP_REGISTRY
|
||||
|
||||
state.status = RunStatus.RUNNING
|
||||
state.save()
|
||||
|
||||
# Resume from the current step — re-execute it so gates
|
||||
# can prompt interactively again.
|
||||
remaining_steps = definition.steps[state.current_step_index :]
|
||||
step_offset = state.current_step_index
|
||||
|
||||
try:
|
||||
self._execute_steps(
|
||||
remaining_steps, context, state, STEP_REGISTRY,
|
||||
step_offset=step_offset,
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
state.status = RunStatus.PAUSED
|
||||
state.append_log({"event": "workflow_interrupted"})
|
||||
state.save()
|
||||
return state
|
||||
except Exception as exc:
|
||||
state.status = RunStatus.FAILED
|
||||
state.append_log({"event": "resume_failed", "error": str(exc)})
|
||||
state.save()
|
||||
raise
|
||||
|
||||
if state.status == RunStatus.RUNNING:
|
||||
state.status = RunStatus.COMPLETED
|
||||
state.append_log({"event": "workflow_finished", "status": state.status.value})
|
||||
state.save()
|
||||
return state
|
||||
|
||||
def _execute_steps(
|
||||
self,
|
||||
steps: list[dict[str, Any]],
|
||||
context: StepContext,
|
||||
state: RunState,
|
||||
registry: dict[str, Any],
|
||||
*,
|
||||
step_offset: int = 0,
|
||||
) -> None:
|
||||
"""Execute a list of steps sequentially."""
|
||||
for i, step_config in enumerate(steps):
|
||||
step_id = step_config.get("id", f"step-{i}")
|
||||
step_type = step_config.get("type", "command")
|
||||
|
||||
state.current_step_id = step_id
|
||||
if step_offset >= 0:
|
||||
state.current_step_index = step_offset + i
|
||||
state.save()
|
||||
|
||||
state.append_log(
|
||||
{"event": "step_started", "step_id": step_id, "type": step_type}
|
||||
)
|
||||
|
||||
# Log progress — use the engine's on_step_start callback if set,
|
||||
# otherwise stay silent (library-safe default).
|
||||
label = step_config.get("command", "") or step_type
|
||||
if self.on_step_start is not None:
|
||||
self.on_step_start(step_id, label)
|
||||
|
||||
step_impl = registry.get(step_type)
|
||||
if not step_impl:
|
||||
state.status = RunStatus.FAILED
|
||||
state.append_log(
|
||||
{
|
||||
"event": "step_failed",
|
||||
"step_id": step_id,
|
||||
"error": f"Unknown step type: {step_type!r}",
|
||||
}
|
||||
)
|
||||
state.save()
|
||||
return
|
||||
|
||||
result: StepResult = step_impl.execute(step_config, context)
|
||||
|
||||
# Record step results — prefer resolved values from step output
|
||||
step_data = {
|
||||
"integration": result.output.get("integration")
|
||||
or step_config.get("integration")
|
||||
or context.default_integration,
|
||||
"model": result.output.get("model")
|
||||
or step_config.get("model")
|
||||
or context.default_model,
|
||||
"options": result.output.get("options")
|
||||
or step_config.get("options", {}),
|
||||
"input": result.output.get("input")
|
||||
or step_config.get("input", {}),
|
||||
"output": result.output,
|
||||
"status": result.status.value,
|
||||
}
|
||||
context.steps[step_id] = step_data
|
||||
state.step_results[step_id] = step_data
|
||||
|
||||
state.append_log(
|
||||
{
|
||||
"event": "step_completed",
|
||||
"step_id": step_id,
|
||||
"status": result.status.value,
|
||||
}
|
||||
)
|
||||
|
||||
# Handle gate pauses
|
||||
if result.status == StepStatus.PAUSED:
|
||||
state.status = RunStatus.PAUSED
|
||||
state.save()
|
||||
return
|
||||
|
||||
# Handle failures
|
||||
if result.status == StepStatus.FAILED:
|
||||
# Gate abort (output.aborted) maps to ABORTED status
|
||||
if result.output.get("aborted"):
|
||||
state.status = RunStatus.ABORTED
|
||||
state.append_log(
|
||||
{
|
||||
"event": "workflow_aborted",
|
||||
"step_id": step_id,
|
||||
}
|
||||
)
|
||||
else:
|
||||
state.status = RunStatus.FAILED
|
||||
state.append_log(
|
||||
{
|
||||
"event": "step_failed",
|
||||
"step_id": step_id,
|
||||
"error": result.error,
|
||||
}
|
||||
)
|
||||
state.save()
|
||||
return
|
||||
|
||||
# Execute nested steps (from control flow)
|
||||
# NOTE: Nested steps run with step_offset=-1 so they don't
|
||||
# update current_step_index. If a nested step pauses,
|
||||
# resume will re-run the parent step and its nested body.
|
||||
# A step-path stack for exact nested resume is a future
|
||||
# enhancement.
|
||||
if result.next_steps:
|
||||
self._execute_steps(
|
||||
result.next_steps, context, state, registry,
|
||||
step_offset=-1,
|
||||
)
|
||||
if state.status in (
|
||||
RunStatus.PAUSED,
|
||||
RunStatus.FAILED,
|
||||
RunStatus.ABORTED,
|
||||
):
|
||||
return
|
||||
|
||||
# Loop iteration: while/do-while re-evaluate after body
|
||||
if step_type in ("while", "do-while"):
|
||||
from .expressions import evaluate_condition
|
||||
|
||||
max_iters = step_config.get("max_iterations")
|
||||
if not isinstance(max_iters, int) or max_iters < 1:
|
||||
max_iters = 10
|
||||
condition = step_config.get("condition", False)
|
||||
for _loop_iter in range(max_iters - 1):
|
||||
if not evaluate_condition(condition, context):
|
||||
break
|
||||
# Namespace nested step IDs per iteration
|
||||
iter_steps = []
|
||||
for ns in result.next_steps:
|
||||
ns_copy = dict(ns)
|
||||
if "id" in ns_copy:
|
||||
ns_copy["id"] = f"{step_id}:{ns_copy['id']}:{_loop_iter + 1}"
|
||||
iter_steps.append(ns_copy)
|
||||
self._execute_steps(
|
||||
iter_steps, context, state, registry,
|
||||
step_offset=-1,
|
||||
)
|
||||
if state.status in (
|
||||
RunStatus.PAUSED,
|
||||
RunStatus.FAILED,
|
||||
RunStatus.ABORTED,
|
||||
):
|
||||
return
|
||||
|
||||
# Fan-out: execute nested step template per item with unique IDs
|
||||
if step_type == "fan-out":
|
||||
items = result.output.get("items", [])
|
||||
template = result.output.get("step_template", {})
|
||||
if template and items:
|
||||
fan_out_results = []
|
||||
for item_idx, item_val in enumerate(result.output["items"]):
|
||||
context.item = item_val
|
||||
# Per-item ID: parentId:templateId:index
|
||||
item_step = dict(template)
|
||||
base_id = item_step.get("id", "item")
|
||||
item_step["id"] = f"{step_id}:{base_id}:{item_idx}"
|
||||
self._execute_steps(
|
||||
[item_step], context, state, registry,
|
||||
step_offset=-1,
|
||||
)
|
||||
# Collect per-item result for fan-in
|
||||
item_result = context.steps.get(item_step["id"], {})
|
||||
fan_out_results.append(item_result.get("output", {}))
|
||||
if state.status in (
|
||||
RunStatus.PAUSED,
|
||||
RunStatus.FAILED,
|
||||
RunStatus.ABORTED,
|
||||
):
|
||||
break
|
||||
context.item = None
|
||||
# Preserve original output and add collected results
|
||||
fan_out_output = dict(result.output)
|
||||
fan_out_output["results"] = fan_out_results
|
||||
context.steps[step_id]["output"] = fan_out_output
|
||||
state.step_results[step_id]["output"] = fan_out_output
|
||||
if state.status in (
|
||||
RunStatus.PAUSED,
|
||||
RunStatus.FAILED,
|
||||
RunStatus.ABORTED,
|
||||
):
|
||||
return
|
||||
else:
|
||||
# Empty items or no template — normalize output
|
||||
result.output["results"] = []
|
||||
context.steps[step_id]["output"] = result.output
|
||||
state.step_results[step_id]["output"] = result.output
|
||||
|
||||
def _resolve_inputs(
|
||||
self,
|
||||
definition: WorkflowDefinition,
|
||||
provided: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Resolve workflow inputs against definitions and provided values."""
|
||||
resolved: dict[str, Any] = {}
|
||||
for name, input_def in definition.inputs.items():
|
||||
if not isinstance(input_def, dict):
|
||||
continue
|
||||
if name in provided:
|
||||
resolved[name] = self._coerce_input(
|
||||
name, provided[name], input_def
|
||||
)
|
||||
elif "default" in input_def:
|
||||
resolved[name] = input_def["default"]
|
||||
elif input_def.get("required", False):
|
||||
msg = f"Required input {name!r} not provided."
|
||||
raise ValueError(msg)
|
||||
return resolved
|
||||
|
||||
@staticmethod
|
||||
def _coerce_input(
|
||||
name: str, value: Any, input_def: dict[str, Any]
|
||||
) -> Any:
|
||||
"""Coerce a provided input value to the declared type."""
|
||||
input_type = input_def.get("type", "string")
|
||||
enum_values = input_def.get("enum")
|
||||
|
||||
if input_type == "number":
|
||||
try:
|
||||
value = float(value)
|
||||
if value == int(value):
|
||||
value = int(value)
|
||||
except (ValueError, TypeError):
|
||||
msg = f"Input {name!r} expected a number, got {value!r}."
|
||||
raise ValueError(msg) from None
|
||||
elif input_type == "boolean":
|
||||
if isinstance(value, str):
|
||||
if value.lower() in ("true", "1", "yes"):
|
||||
value = True
|
||||
elif value.lower() in ("false", "0", "no"):
|
||||
value = False
|
||||
else:
|
||||
msg = f"Input {name!r} expected a boolean, got {value!r}."
|
||||
raise ValueError(msg)
|
||||
|
||||
if enum_values is not None and value not in enum_values:
|
||||
msg = (
|
||||
f"Input {name!r} value {value!r} not in allowed "
|
||||
f"values: {enum_values}."
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
return value
|
||||
|
||||
def list_runs(self) -> list[dict[str, Any]]:
|
||||
"""List all workflow runs in the project."""
|
||||
runs_dir = self.project_root / ".specify" / "workflows" / "runs"
|
||||
if not runs_dir.exists():
|
||||
return []
|
||||
|
||||
runs: list[dict[str, Any]] = []
|
||||
for run_dir in sorted(runs_dir.iterdir()):
|
||||
if not run_dir.is_dir():
|
||||
continue
|
||||
state_path = run_dir / "state.json"
|
||||
if state_path.exists():
|
||||
with open(state_path, encoding="utf-8") as f:
|
||||
state_data = json.load(f)
|
||||
runs.append(state_data)
|
||||
return runs
|
||||
|
||||
|
||||
class WorkflowAbortError(Exception):
|
||||
"""Raised when a workflow is aborted (e.g., gate rejection)."""
|
||||
300
src/specify_cli/workflows/expressions.py
Normal file
300
src/specify_cli/workflows/expressions.py
Normal file
@@ -0,0 +1,300 @@
|
||||
"""Sandboxed expression evaluator for workflow templates.
|
||||
|
||||
Provides a safe Jinja2 subset for evaluating expressions in workflow YAML.
|
||||
No file I/O, no imports, no arbitrary code execution.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
|
||||
# -- Custom filters -------------------------------------------------------
|
||||
|
||||
def _filter_default(value: Any, default_value: Any = "") -> Any:
|
||||
"""Return *default_value* when *value* is ``None`` or empty string."""
|
||||
if value is None or value == "":
|
||||
return default_value
|
||||
return value
|
||||
|
||||
|
||||
def _filter_join(value: Any, separator: str = ", ") -> str:
|
||||
"""Join a list into a string with *separator*."""
|
||||
if isinstance(value, list):
|
||||
return separator.join(str(v) for v in value)
|
||||
return str(value)
|
||||
|
||||
|
||||
def _filter_map(value: Any, attr: str) -> list[Any]:
|
||||
"""Map a list of dicts to a specific attribute."""
|
||||
if isinstance(value, list):
|
||||
result = []
|
||||
for item in value:
|
||||
if isinstance(item, dict):
|
||||
# Support dot notation: "result.status" → item["result"]["status"]
|
||||
parts = attr.split(".")
|
||||
v = item
|
||||
for part in parts:
|
||||
if isinstance(v, dict):
|
||||
v = v.get(part)
|
||||
else:
|
||||
v = None
|
||||
break
|
||||
result.append(v)
|
||||
else:
|
||||
result.append(item)
|
||||
return result
|
||||
return []
|
||||
|
||||
|
||||
def _filter_contains(value: Any, substring: str) -> bool:
|
||||
"""Check if a string or list contains *substring*."""
|
||||
if isinstance(value, str):
|
||||
return substring in value
|
||||
if isinstance(value, list):
|
||||
return substring in value
|
||||
return False
|
||||
|
||||
|
||||
# -- Expression resolution ------------------------------------------------
|
||||
|
||||
_EXPR_PATTERN = re.compile(r"\{\{(.+?)\}\}")
|
||||
|
||||
|
||||
def _resolve_dot_path(obj: Any, path: str) -> Any:
|
||||
"""Resolve a dotted path like ``steps.specify.output.file`` against *obj*.
|
||||
|
||||
Supports dict key access and list indexing (e.g., ``task_list[0]``).
|
||||
"""
|
||||
parts = path.split(".")
|
||||
current = obj
|
||||
for part in parts:
|
||||
# Handle list indexing: name[0]
|
||||
idx_match = re.match(r"^([\w-]+)\[(\d+)\]$", part)
|
||||
if idx_match:
|
||||
key, idx = idx_match.group(1), int(idx_match.group(2))
|
||||
if isinstance(current, dict):
|
||||
current = current.get(key)
|
||||
else:
|
||||
return None
|
||||
if isinstance(current, list) and 0 <= idx < len(current):
|
||||
current = current[idx]
|
||||
else:
|
||||
return None
|
||||
elif isinstance(current, dict):
|
||||
current = current.get(part)
|
||||
else:
|
||||
return None
|
||||
if current is None:
|
||||
return None
|
||||
return current
|
||||
|
||||
|
||||
def _build_namespace(context: Any) -> dict[str, Any]:
|
||||
"""Build the variable namespace from a StepContext."""
|
||||
ns: dict[str, Any] = {}
|
||||
if hasattr(context, "inputs"):
|
||||
ns["inputs"] = context.inputs or {}
|
||||
if hasattr(context, "steps"):
|
||||
ns["steps"] = context.steps or {}
|
||||
if hasattr(context, "item"):
|
||||
ns["item"] = context.item
|
||||
if hasattr(context, "fan_in"):
|
||||
ns["fan_in"] = context.fan_in or {}
|
||||
return ns
|
||||
|
||||
|
||||
def _evaluate_simple_expression(expr: str, namespace: dict[str, Any]) -> Any:
|
||||
"""Evaluate a simple expression against the namespace.
|
||||
|
||||
Supports:
|
||||
- Dot-path access: ``steps.specify.output.file``
|
||||
- Comparisons: ``==``, ``!=``, ``>``, ``<``, ``>=``, ``<=``
|
||||
- Boolean operators: ``and``, ``or``, ``not``
|
||||
- ``in``, ``not in``
|
||||
- Pipe filters: ``| default('...')``, ``| join(', ')``, ``| contains('...')``, ``| map('...')``
|
||||
- String and numeric literals
|
||||
"""
|
||||
expr = expr.strip()
|
||||
|
||||
# String literal — check before pipes and operators so quoted strings
|
||||
# containing | or operator keywords are not mis-parsed.
|
||||
if (expr.startswith("'") and expr.endswith("'")) or (
|
||||
expr.startswith('"') and expr.endswith('"')
|
||||
):
|
||||
return expr[1:-1]
|
||||
|
||||
# Handle pipe filters
|
||||
if "|" in expr:
|
||||
parts = expr.split("|", 1)
|
||||
value = _evaluate_simple_expression(parts[0].strip(), namespace)
|
||||
filter_expr = parts[1].strip()
|
||||
|
||||
# Parse filter name and argument
|
||||
filter_match = re.match(r"(\w+)\((.+)\)", filter_expr)
|
||||
if filter_match:
|
||||
fname = filter_match.group(1)
|
||||
farg = _evaluate_simple_expression(filter_match.group(2).strip(), namespace)
|
||||
if fname == "default":
|
||||
return _filter_default(value, farg)
|
||||
if fname == "join":
|
||||
return _filter_join(value, farg)
|
||||
if fname == "map":
|
||||
return _filter_map(value, farg)
|
||||
if fname == "contains":
|
||||
return _filter_contains(value, farg)
|
||||
# Filter without args
|
||||
filter_name = filter_expr.strip()
|
||||
if filter_name == "default":
|
||||
return _filter_default(value)
|
||||
return value
|
||||
|
||||
# Boolean operators — parse 'or' first (lower precedence) so that
|
||||
# 'a or b and c' is evaluated as 'a or (b and c)'.
|
||||
if " or " in expr:
|
||||
parts = expr.split(" or ", 1)
|
||||
left = _evaluate_simple_expression(parts[0].strip(), namespace)
|
||||
right = _evaluate_simple_expression(parts[1].strip(), namespace)
|
||||
return bool(left) or bool(right)
|
||||
|
||||
if " and " in expr:
|
||||
parts = expr.split(" and ", 1)
|
||||
left = _evaluate_simple_expression(parts[0].strip(), namespace)
|
||||
right = _evaluate_simple_expression(parts[1].strip(), namespace)
|
||||
return bool(left) and bool(right)
|
||||
|
||||
if expr.startswith("not "):
|
||||
inner = _evaluate_simple_expression(expr[4:].strip(), namespace)
|
||||
return not bool(inner)
|
||||
|
||||
# Comparison operators (order matters — check multi-char ops first)
|
||||
for op in ("!=", "==", ">=", "<=", ">", "<", " not in ", " in "):
|
||||
if op in expr:
|
||||
parts = expr.split(op, 1)
|
||||
left = _evaluate_simple_expression(parts[0].strip(), namespace)
|
||||
right = _evaluate_simple_expression(parts[1].strip(), namespace)
|
||||
if op == "==":
|
||||
return left == right
|
||||
if op == "!=":
|
||||
return left != right
|
||||
if op == ">":
|
||||
return _safe_compare(left, right, ">")
|
||||
if op == "<":
|
||||
return _safe_compare(left, right, "<")
|
||||
if op == ">=":
|
||||
return _safe_compare(left, right, ">=")
|
||||
if op == "<=":
|
||||
return _safe_compare(left, right, "<=")
|
||||
if op == " in ":
|
||||
return left in right if right is not None else False
|
||||
if op == " not in ":
|
||||
return left not in right if right is not None else True
|
||||
|
||||
# Numeric literal
|
||||
try:
|
||||
if "." in expr:
|
||||
return float(expr)
|
||||
return int(expr)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Boolean literal
|
||||
if expr.lower() == "true":
|
||||
return True
|
||||
if expr.lower() == "false":
|
||||
return False
|
||||
|
||||
# Null
|
||||
if expr.lower() in ("none", "null"):
|
||||
return None
|
||||
|
||||
# List literal (simple)
|
||||
if expr.startswith("[") and expr.endswith("]"):
|
||||
inner = expr[1:-1].strip()
|
||||
if not inner:
|
||||
return []
|
||||
items = [_evaluate_simple_expression(i.strip(), namespace) for i in inner.split(",")]
|
||||
return items
|
||||
|
||||
# Variable reference (dot-path)
|
||||
return _resolve_dot_path(namespace, expr)
|
||||
|
||||
|
||||
def _safe_compare(left: Any, right: Any, op: str) -> bool:
|
||||
"""Safely compare two values, coercing types when possible."""
|
||||
try:
|
||||
if isinstance(left, str):
|
||||
left = float(left) if "." in left else int(left)
|
||||
if isinstance(right, str):
|
||||
right = float(right) if "." in right else int(right)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
try:
|
||||
if op == ">":
|
||||
return left > right # type: ignore[operator]
|
||||
if op == "<":
|
||||
return left < right # type: ignore[operator]
|
||||
if op == ">=":
|
||||
return left >= right # type: ignore[operator]
|
||||
if op == "<=":
|
||||
return left <= right # type: ignore[operator]
|
||||
except TypeError:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def evaluate_expression(template: str, context: Any) -> Any:
|
||||
"""Evaluate a template string with ``{{ ... }}`` expressions.
|
||||
|
||||
If the entire string is a single expression, returns the raw value
|
||||
(preserving type). Otherwise, substitutes each expression inline
|
||||
and returns a string.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
template:
|
||||
The template string (e.g., ``"{{ steps.plan.output.task_count }}"``
|
||||
or ``"Processed {{ inputs.feature_name }}"``.
|
||||
context:
|
||||
A ``StepContext`` or compatible object.
|
||||
|
||||
Returns
|
||||
-------
|
||||
The resolved value (any type for single-expression templates,
|
||||
string for multi-expression or mixed templates).
|
||||
"""
|
||||
if not isinstance(template, str):
|
||||
return template
|
||||
|
||||
namespace = _build_namespace(context)
|
||||
|
||||
# Single expression: return typed value
|
||||
match = _EXPR_PATTERN.fullmatch(template.strip())
|
||||
if match:
|
||||
return _evaluate_simple_expression(match.group(1).strip(), namespace)
|
||||
|
||||
# Multi-expression: string interpolation
|
||||
def _replacer(m: re.Match[str]) -> str:
|
||||
val = _evaluate_simple_expression(m.group(1).strip(), namespace)
|
||||
return str(val) if val is not None else ""
|
||||
|
||||
return _EXPR_PATTERN.sub(_replacer, template)
|
||||
|
||||
|
||||
def evaluate_condition(condition: str, context: Any) -> bool:
|
||||
"""Evaluate a condition expression and return a boolean.
|
||||
|
||||
Convenience wrapper around ``evaluate_expression`` that coerces
|
||||
the result to bool.
|
||||
"""
|
||||
result = evaluate_expression(condition, context)
|
||||
# Treat plain "false"/"true" strings as booleans so that
|
||||
# condition: "false" (without {{ }}) behaves as expected.
|
||||
if isinstance(result, str):
|
||||
lower = result.lower()
|
||||
if lower == "false":
|
||||
return False
|
||||
if lower == "true":
|
||||
return True
|
||||
return bool(result)
|
||||
1
src/specify_cli/workflows/steps/__init__.py
Normal file
1
src/specify_cli/workflows/steps/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Auto-discovery for built-in step types."""
|
||||
155
src/specify_cli/workflows/steps/command/__init__.py
Normal file
155
src/specify_cli/workflows/steps/command/__init__.py
Normal file
@@ -0,0 +1,155 @@
|
||||
"""Command step — dispatches a Spec Kit command to an integration CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class CommandStep(StepBase):
|
||||
"""Default step type — invokes a Spec Kit command via the integration CLI.
|
||||
|
||||
The command files (skills, markdown, TOML) are already installed in
|
||||
the integration's directory on disk. This step tells the CLI to
|
||||
execute the command by name (e.g. ``/speckit.specify`` or
|
||||
``/speckit-specify``) rather than reading the file contents.
|
||||
|
||||
.. note::
|
||||
|
||||
CLI output is streamed to the terminal for live progress.
|
||||
``output.exit_code`` is always captured and can be referenced
|
||||
by later steps (e.g. ``{{ steps.specify.output.exit_code }}``).
|
||||
Full ``stdout``/``stderr`` capture is a planned enhancement.
|
||||
"""
|
||||
|
||||
type_key = "command"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
command = config.get("command", "")
|
||||
input_data = config.get("input", {})
|
||||
|
||||
# Resolve expressions in input
|
||||
resolved_input: dict[str, Any] = {}
|
||||
for key, value in input_data.items():
|
||||
resolved_input[key] = evaluate_expression(value, context)
|
||||
|
||||
# Resolve integration (step → workflow default → project default)
|
||||
integration = config.get("integration") or context.default_integration
|
||||
if integration and isinstance(integration, str) and "{{" in integration:
|
||||
integration = evaluate_expression(integration, context)
|
||||
|
||||
# Resolve model
|
||||
model = config.get("model") or context.default_model
|
||||
if model and isinstance(model, str) and "{{" in model:
|
||||
model = evaluate_expression(model, context)
|
||||
|
||||
# Merge options (workflow defaults ← step overrides)
|
||||
options = dict(context.default_options)
|
||||
step_options = config.get("options", {})
|
||||
if step_options:
|
||||
options.update(step_options)
|
||||
|
||||
# Attempt CLI dispatch
|
||||
args_str = str(resolved_input.get("args", ""))
|
||||
dispatch_result = self._try_dispatch(
|
||||
command, integration, model, args_str, context
|
||||
)
|
||||
|
||||
output: dict[str, Any] = {
|
||||
"command": command,
|
||||
"integration": integration,
|
||||
"model": model,
|
||||
"options": options,
|
||||
"input": resolved_input,
|
||||
}
|
||||
|
||||
if dispatch_result is not None:
|
||||
output["exit_code"] = dispatch_result["exit_code"]
|
||||
output["stdout"] = dispatch_result["stdout"]
|
||||
output["stderr"] = dispatch_result["stderr"]
|
||||
output["dispatched"] = True
|
||||
if dispatch_result["exit_code"] != 0:
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
output=output,
|
||||
error=dispatch_result["stderr"] or f"Command exited with code {dispatch_result['exit_code']}",
|
||||
)
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output=output,
|
||||
)
|
||||
else:
|
||||
output["exit_code"] = 1
|
||||
output["dispatched"] = False
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
output=output,
|
||||
error=(
|
||||
f"Cannot dispatch command {command!r}: "
|
||||
f"integration {integration!r} CLI not found or not installed. "
|
||||
f"Install the CLI tool or check 'specify integration list'."
|
||||
),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _try_dispatch(
|
||||
command: str,
|
||||
integration_key: str | None,
|
||||
model: str | None,
|
||||
args: str,
|
||||
context: StepContext,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Invoke *command* by name through the integration CLI.
|
||||
|
||||
The integration's ``dispatch_command`` builds the native
|
||||
slash-command invocation (e.g. ``/speckit.specify`` for
|
||||
markdown agents, ``/speckit-specify`` for skills agents),
|
||||
then executes the CLI non-interactively.
|
||||
|
||||
Returns the dispatch result dict, or ``None`` if dispatch is
|
||||
not possible (integration not found, CLI not installed, or
|
||||
dispatch not supported).
|
||||
"""
|
||||
if not integration_key:
|
||||
return None
|
||||
|
||||
try:
|
||||
from specify_cli.integrations import get_integration
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
impl = get_integration(integration_key)
|
||||
if impl is None:
|
||||
return None
|
||||
|
||||
# Check if the integration supports CLI dispatch
|
||||
if impl.build_exec_args("test") is None:
|
||||
return None
|
||||
|
||||
# Check if the CLI tool is actually installed
|
||||
if not shutil.which(impl.key):
|
||||
return None
|
||||
|
||||
project_root = Path(context.project_root) if context.project_root else None
|
||||
|
||||
try:
|
||||
return impl.dispatch_command(
|
||||
command,
|
||||
args=args,
|
||||
project_root=project_root,
|
||||
model=model,
|
||||
)
|
||||
except (NotImplementedError, OSError):
|
||||
return None
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "command" not in config:
|
||||
errors.append(
|
||||
f"Command step {config.get('id', '?')!r} is missing 'command' field."
|
||||
)
|
||||
return errors
|
||||
61
src/specify_cli/workflows/steps/do_while/__init__.py
Normal file
61
src/specify_cli/workflows/steps/do_while/__init__.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""Do-While loop step — execute at least once, then repeat while condition is truthy."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
|
||||
|
||||
class DoWhileStep(StepBase):
|
||||
"""Execute body at least once, then check condition.
|
||||
|
||||
Continues while condition is truthy. ``max_iterations`` is an
|
||||
optional safety cap (defaults to 10 if omitted).
|
||||
|
||||
The first invocation always returns the nested steps for execution.
|
||||
The engine re-evaluates ``step_config['condition']`` after each
|
||||
iteration to decide whether to loop again.
|
||||
"""
|
||||
|
||||
type_key = "do-while"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
max_iterations = config.get("max_iterations")
|
||||
if max_iterations is None:
|
||||
max_iterations = 10
|
||||
nested_steps = config.get("steps", [])
|
||||
condition = config.get("condition", "false")
|
||||
|
||||
# Always execute body at least once; the engine layer evaluates
|
||||
# `condition` after each iteration to decide whether to loop.
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={
|
||||
"condition": condition,
|
||||
"max_iterations": max_iterations,
|
||||
"loop_type": "do-while",
|
||||
},
|
||||
next_steps=nested_steps,
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "condition" not in config:
|
||||
errors.append(
|
||||
f"Do-while step {config.get('id', '?')!r} is missing "
|
||||
f"'condition' field."
|
||||
)
|
||||
max_iter = config.get("max_iterations")
|
||||
if max_iter is not None:
|
||||
if not isinstance(max_iter, int) or max_iter < 1:
|
||||
errors.append(
|
||||
f"Do-while step {config.get('id', '?')!r}: "
|
||||
f"'max_iterations' must be an integer >= 1."
|
||||
)
|
||||
nested = config.get("steps", [])
|
||||
if not isinstance(nested, list):
|
||||
errors.append(
|
||||
f"Do-while step {config.get('id', '?')!r}: 'steps' must be a list."
|
||||
)
|
||||
return errors
|
||||
61
src/specify_cli/workflows/steps/fan_in/__init__.py
Normal file
61
src/specify_cli/workflows/steps/fan_in/__init__.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""Fan-in step — join point for parallel steps."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class FanInStep(StepBase):
|
||||
"""Join point that aggregates results from ``wait_for:`` steps.
|
||||
|
||||
Reads completed step outputs from ``context.steps`` and collects
|
||||
them into ``output.results``. Does not block; relies on the
|
||||
engine executing steps sequentially.
|
||||
"""
|
||||
|
||||
type_key = "fan-in"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
wait_for = config.get("wait_for", [])
|
||||
output_config = config.get("output") or {}
|
||||
if not isinstance(output_config, dict):
|
||||
output_config = {}
|
||||
|
||||
# Collect results from referenced steps
|
||||
results = []
|
||||
for step_id in wait_for:
|
||||
step_data = context.steps.get(step_id, {})
|
||||
results.append(step_data.get("output", {}))
|
||||
|
||||
# Resolve output expressions with fan_in in context
|
||||
prev_fan_in = getattr(context, "fan_in", None)
|
||||
context.fan_in = {"results": results}
|
||||
resolved_output: dict[str, Any] = {"results": results}
|
||||
|
||||
try:
|
||||
for key, expr in output_config.items():
|
||||
if isinstance(expr, str) and "{{" in expr:
|
||||
resolved_output[key] = evaluate_expression(expr, context)
|
||||
else:
|
||||
resolved_output[key] = expr
|
||||
finally:
|
||||
# Restore previous fan_in state even if evaluation fails
|
||||
context.fan_in = prev_fan_in
|
||||
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output=resolved_output,
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
wait_for = config.get("wait_for", [])
|
||||
if not isinstance(wait_for, list) or not wait_for:
|
||||
errors.append(
|
||||
f"Fan-in step {config.get('id', '?')!r}: "
|
||||
f"'wait_for' must be a non-empty list of step IDs."
|
||||
)
|
||||
return errors
|
||||
58
src/specify_cli/workflows/steps/fan_out/__init__.py
Normal file
58
src/specify_cli/workflows/steps/fan_out/__init__.py
Normal file
@@ -0,0 +1,58 @@
|
||||
"""Fan-out step — dispatch a step template over a collection."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class FanOutStep(StepBase):
|
||||
"""Dispatch a step template for each item in a collection.
|
||||
|
||||
The engine executes the nested ``step:`` template once per item,
|
||||
setting ``context.item`` for each iteration. Execution is
|
||||
currently sequential; ``max_concurrency`` is accepted but not
|
||||
enforced.
|
||||
"""
|
||||
|
||||
type_key = "fan-out"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
items_expr = config.get("items", "[]")
|
||||
items = evaluate_expression(items_expr, context)
|
||||
if not isinstance(items, list):
|
||||
items = []
|
||||
|
||||
max_concurrency = config.get("max_concurrency", 1)
|
||||
step_template = config.get("step", {})
|
||||
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={
|
||||
"items": items,
|
||||
"max_concurrency": max_concurrency,
|
||||
"step_template": step_template,
|
||||
"item_count": len(items),
|
||||
},
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "items" not in config:
|
||||
errors.append(
|
||||
f"Fan-out step {config.get('id', '?')!r} is missing "
|
||||
f"'items' field."
|
||||
)
|
||||
if "step" not in config:
|
||||
errors.append(
|
||||
f"Fan-out step {config.get('id', '?')!r} is missing "
|
||||
f"'step' field (nested step template)."
|
||||
)
|
||||
step = config.get("step")
|
||||
if step is not None and not isinstance(step, dict):
|
||||
errors.append(
|
||||
f"Fan-out step {config.get('id', '?')!r}: 'step' must be a mapping."
|
||||
)
|
||||
return errors
|
||||
121
src/specify_cli/workflows/steps/gate/__init__.py
Normal file
121
src/specify_cli/workflows/steps/gate/__init__.py
Normal file
@@ -0,0 +1,121 @@
|
||||
"""Gate step — human review gate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class GateStep(StepBase):
|
||||
"""Interactive review gate.
|
||||
|
||||
When running in an interactive terminal, prompts the user to choose
|
||||
an option (e.g. approve / reject). Falls back to ``PAUSED`` when
|
||||
stdin is not a TTY (CI, piped input) so the run can be resumed
|
||||
later with ``specify workflow resume``.
|
||||
|
||||
The user's choice is stored in ``output.choice``. ``on_reject``
|
||||
controls abort / skip behaviour.
|
||||
"""
|
||||
|
||||
type_key = "gate"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
message = config.get("message", "Review required.")
|
||||
if isinstance(message, str) and "{{" in message:
|
||||
message = evaluate_expression(message, context)
|
||||
|
||||
options = config.get("options", ["approve", "reject"])
|
||||
on_reject = config.get("on_reject", "abort")
|
||||
|
||||
show_file = config.get("show_file")
|
||||
if show_file and isinstance(show_file, str) and "{{" in show_file:
|
||||
show_file = evaluate_expression(show_file, context)
|
||||
|
||||
output = {
|
||||
"message": message,
|
||||
"options": options,
|
||||
"on_reject": on_reject,
|
||||
"show_file": show_file,
|
||||
"choice": None,
|
||||
}
|
||||
|
||||
# Non-interactive: pause for later resume
|
||||
if not sys.stdin.isatty():
|
||||
return StepResult(status=StepStatus.PAUSED, output=output)
|
||||
|
||||
# Interactive: prompt the user
|
||||
choice = self._prompt(message, options)
|
||||
output["choice"] = choice
|
||||
|
||||
if choice in ("reject", "abort"):
|
||||
if on_reject == "abort":
|
||||
output["aborted"] = True
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
output=output,
|
||||
error=f"Gate rejected by user at step {config.get('id', '?')!r}",
|
||||
)
|
||||
if on_reject == "retry":
|
||||
# Pause so the next resume re-executes this gate
|
||||
return StepResult(status=StepStatus.PAUSED, output=output)
|
||||
# on_reject == "skip" → completed, downstream steps decide
|
||||
return StepResult(status=StepStatus.COMPLETED, output=output)
|
||||
|
||||
return StepResult(status=StepStatus.COMPLETED, output=output)
|
||||
|
||||
@staticmethod
|
||||
def _prompt(message: str, options: list[str]) -> str:
|
||||
"""Display gate message and prompt for a choice."""
|
||||
print("\n ┌─ Gate ─────────────────────────────────────")
|
||||
print(f" │ {message}")
|
||||
print(" │")
|
||||
for i, opt in enumerate(options, 1):
|
||||
print(f" │ [{i}] {opt}")
|
||||
print(" └────────────────────────────────────────────")
|
||||
|
||||
while True:
|
||||
try:
|
||||
raw = input(f" Choose [1-{len(options)}]: ").strip()
|
||||
except (EOFError, KeyboardInterrupt):
|
||||
print()
|
||||
return options[-1] # default to last (usually reject)
|
||||
if raw.isdigit() and 1 <= int(raw) <= len(options):
|
||||
return options[int(raw) - 1]
|
||||
# Also accept the option name directly
|
||||
if raw.lower() in [o.lower() for o in options]:
|
||||
return next(o for o in options if o.lower() == raw.lower())
|
||||
print(f" Invalid choice. Enter 1-{len(options)} or an option name.")
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "message" not in config:
|
||||
errors.append(
|
||||
f"Gate step {config.get('id', '?')!r} is missing 'message' field."
|
||||
)
|
||||
options = config.get("options", ["approve", "reject"])
|
||||
if not isinstance(options, list) or not options:
|
||||
errors.append(
|
||||
f"Gate step {config.get('id', '?')!r}: 'options' must be a non-empty list."
|
||||
)
|
||||
elif not all(isinstance(o, str) for o in options):
|
||||
errors.append(
|
||||
f"Gate step {config.get('id', '?')!r}: all options must be strings."
|
||||
)
|
||||
on_reject = config.get("on_reject", "abort")
|
||||
if on_reject not in ("abort", "skip", "retry"):
|
||||
errors.append(
|
||||
f"Gate step {config.get('id', '?')!r}: 'on_reject' must be "
|
||||
f"'abort', 'skip', or 'retry'."
|
||||
)
|
||||
if on_reject in ("abort", "retry") and isinstance(options, list):
|
||||
reject_choices = {"reject", "abort"}
|
||||
if not any(o.lower() in reject_choices for o in options):
|
||||
errors.append(
|
||||
f"Gate step {config.get('id', '?')!r}: on_reject={on_reject!r} "
|
||||
f"but options has no 'reject' or 'abort' choice."
|
||||
)
|
||||
return errors
|
||||
55
src/specify_cli/workflows/steps/if_then/__init__.py
Normal file
55
src/specify_cli/workflows/steps/if_then/__init__.py
Normal file
@@ -0,0 +1,55 @@
|
||||
"""If/Then/Else step — conditional branching."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_condition
|
||||
|
||||
|
||||
class IfThenStep(StepBase):
|
||||
"""Branch based on a boolean condition expression.
|
||||
|
||||
Both ``then:`` and ``else:`` contain inline step arrays — full step
|
||||
definitions, not ID references.
|
||||
"""
|
||||
|
||||
type_key = "if"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
condition = config.get("condition", False)
|
||||
result = evaluate_condition(condition, context)
|
||||
|
||||
if result:
|
||||
branch = config.get("then", [])
|
||||
else:
|
||||
branch = config.get("else", [])
|
||||
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={"condition_result": result},
|
||||
next_steps=branch,
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "condition" not in config:
|
||||
errors.append(
|
||||
f"If step {config.get('id', '?')!r} is missing 'condition' field."
|
||||
)
|
||||
if "then" not in config:
|
||||
errors.append(
|
||||
f"If step {config.get('id', '?')!r} is missing 'then' field."
|
||||
)
|
||||
then_branch = config.get("then", [])
|
||||
if not isinstance(then_branch, list):
|
||||
errors.append(
|
||||
f"If step {config.get('id', '?')!r}: 'then' must be a list of steps."
|
||||
)
|
||||
else_branch = config.get("else", [])
|
||||
if else_branch and not isinstance(else_branch, list):
|
||||
errors.append(
|
||||
f"If step {config.get('id', '?')!r}: 'else' must be a list of steps."
|
||||
)
|
||||
return errors
|
||||
156
src/specify_cli/workflows/steps/prompt/__init__.py
Normal file
156
src/specify_cli/workflows/steps/prompt/__init__.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""Prompt step — sends an arbitrary prompt to an integration CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class PromptStep(StepBase):
|
||||
"""Send a free-form prompt to an integration CLI.
|
||||
|
||||
Unlike ``CommandStep`` which invokes an installed Spec Kit command
|
||||
by name (e.g. ``/speckit.specify`` or ``/speckit-specify``),
|
||||
``PromptStep`` sends an arbitrary inline ``prompt:`` string
|
||||
directly to the CLI. This is useful for ad-hoc instructions
|
||||
that don't map to a registered command.
|
||||
|
||||
.. note::
|
||||
|
||||
CLI output is streamed to the terminal for live progress.
|
||||
``output.exit_code`` is always captured and can be referenced
|
||||
by later steps. Full response text capture is a planned
|
||||
enhancement.
|
||||
|
||||
Example YAML::
|
||||
|
||||
- id: review-security
|
||||
type: prompt
|
||||
prompt: "Review {{ inputs.file }} for security vulnerabilities"
|
||||
integration: claude
|
||||
"""
|
||||
|
||||
type_key = "prompt"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
prompt_template = config.get("prompt", "")
|
||||
prompt = evaluate_expression(prompt_template, context)
|
||||
if not isinstance(prompt, str):
|
||||
prompt = str(prompt)
|
||||
|
||||
# Resolve integration (step → workflow default)
|
||||
integration = config.get("integration") or context.default_integration
|
||||
if integration and isinstance(integration, str) and "{{" in integration:
|
||||
integration = evaluate_expression(integration, context)
|
||||
|
||||
# Resolve model
|
||||
model = config.get("model") or context.default_model
|
||||
if model and isinstance(model, str) and "{{" in model:
|
||||
model = evaluate_expression(model, context)
|
||||
|
||||
# Attempt CLI dispatch
|
||||
dispatch_result = self._try_dispatch(
|
||||
prompt, integration, model, context
|
||||
)
|
||||
|
||||
output: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"integration": integration,
|
||||
"model": model,
|
||||
}
|
||||
|
||||
if dispatch_result is not None:
|
||||
output["exit_code"] = dispatch_result["exit_code"]
|
||||
output["stdout"] = dispatch_result["stdout"]
|
||||
output["stderr"] = dispatch_result["stderr"]
|
||||
output["dispatched"] = True
|
||||
if dispatch_result["exit_code"] != 0:
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
output=output,
|
||||
error=(
|
||||
dispatch_result["stderr"]
|
||||
or f"Prompt exited with code {dispatch_result['exit_code']}"
|
||||
),
|
||||
)
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output=output,
|
||||
)
|
||||
else:
|
||||
output["exit_code"] = 1
|
||||
output["dispatched"] = False
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
output=output,
|
||||
error=(
|
||||
f"Cannot dispatch prompt: "
|
||||
f"integration {integration!r} "
|
||||
f"CLI not found or not installed."
|
||||
),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _try_dispatch(
|
||||
prompt: str,
|
||||
integration_key: str | None,
|
||||
model: str | None,
|
||||
context: StepContext,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Dispatch *prompt* directly through the integration CLI."""
|
||||
if not integration_key or not prompt:
|
||||
return None
|
||||
|
||||
try:
|
||||
from specify_cli.integrations import get_integration
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
impl = get_integration(integration_key)
|
||||
if impl is None:
|
||||
return None
|
||||
|
||||
exec_args = impl.build_exec_args(prompt, model=model, output_json=False)
|
||||
if exec_args is None:
|
||||
return None
|
||||
|
||||
if not shutil.which(impl.key):
|
||||
return None
|
||||
|
||||
import subprocess
|
||||
|
||||
project_root = (
|
||||
Path(context.project_root) if context.project_root else Path.cwd()
|
||||
)
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
exec_args,
|
||||
text=True,
|
||||
cwd=str(project_root),
|
||||
)
|
||||
return {
|
||||
"exit_code": result.returncode,
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
}
|
||||
except KeyboardInterrupt:
|
||||
return {
|
||||
"exit_code": 130,
|
||||
"stdout": "",
|
||||
"stderr": "Interrupted by user",
|
||||
}
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "prompt" not in config:
|
||||
errors.append(
|
||||
f"Prompt step {config.get('id', '?')!r} is missing 'prompt' field."
|
||||
)
|
||||
return errors
|
||||
75
src/specify_cli/workflows/steps/shell/__init__.py
Normal file
75
src/specify_cli/workflows/steps/shell/__init__.py
Normal file
@@ -0,0 +1,75 @@
|
||||
"""Shell step — run a local shell command."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class ShellStep(StepBase):
|
||||
"""Run a local shell command (non-agent).
|
||||
|
||||
Captures exit code and stdout/stderr.
|
||||
"""
|
||||
|
||||
type_key = "shell"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
run_cmd = config.get("run", "")
|
||||
if isinstance(run_cmd, str) and "{{" in run_cmd:
|
||||
run_cmd = evaluate_expression(run_cmd, context)
|
||||
run_cmd = str(run_cmd)
|
||||
|
||||
cwd = context.project_root or "."
|
||||
|
||||
# NOTE: shell=True is required to support pipes, redirects, and
|
||||
# multi-command expressions in workflow YAML. Workflow authors
|
||||
# control commands; catalog-installed workflows should be reviewed
|
||||
# before use (see PUBLISHING.md for security guidance).
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
run_cmd,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=cwd,
|
||||
timeout=300,
|
||||
)
|
||||
output = {
|
||||
"exit_code": proc.returncode,
|
||||
"stdout": proc.stdout,
|
||||
"stderr": proc.stderr,
|
||||
}
|
||||
if proc.returncode != 0:
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
error=f"Shell command exited with code {proc.returncode}.",
|
||||
output=output,
|
||||
)
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output=output,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
error="Shell command timed out after 300 seconds.",
|
||||
output={"exit_code": -1, "stdout": "", "stderr": "timeout"},
|
||||
)
|
||||
except OSError as exc:
|
||||
return StepResult(
|
||||
status=StepStatus.FAILED,
|
||||
error=f"Shell command failed: {exc}",
|
||||
output={"exit_code": -1, "stdout": "", "stderr": str(exc)},
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "run" not in config:
|
||||
errors.append(
|
||||
f"Shell step {config.get('id', '?')!r} is missing 'run' field."
|
||||
)
|
||||
return errors
|
||||
70
src/specify_cli/workflows/steps/switch/__init__.py
Normal file
70
src/specify_cli/workflows/steps/switch/__init__.py
Normal file
@@ -0,0 +1,70 @@
|
||||
"""Switch step — multi-branch dispatch."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_expression
|
||||
|
||||
|
||||
class SwitchStep(StepBase):
|
||||
"""Multi-branch dispatch on an expression.
|
||||
|
||||
Evaluates ``expression:`` once, matches against ``cases:`` keys
|
||||
(exact match, string-coerced). Falls through to ``default:`` if
|
||||
no case matches.
|
||||
"""
|
||||
|
||||
type_key = "switch"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
expression = config.get("expression", "")
|
||||
value = evaluate_expression(expression, context)
|
||||
|
||||
# String-coerce for matching
|
||||
str_value = str(value) if value is not None else ""
|
||||
|
||||
cases = config.get("cases", {})
|
||||
for case_key, case_steps in cases.items():
|
||||
if str(case_key) == str_value:
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={"matched_case": str(case_key), "expression_value": value},
|
||||
next_steps=case_steps,
|
||||
)
|
||||
|
||||
# Default fallback
|
||||
default_steps = config.get("default", [])
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={"matched_case": "__default__", "expression_value": value},
|
||||
next_steps=default_steps,
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "expression" not in config:
|
||||
errors.append(
|
||||
f"Switch step {config.get('id', '?')!r} is missing "
|
||||
f"'expression' field."
|
||||
)
|
||||
cases = config.get("cases", {})
|
||||
if not isinstance(cases, dict):
|
||||
errors.append(
|
||||
f"Switch step {config.get('id', '?')!r}: 'cases' must be a mapping."
|
||||
)
|
||||
else:
|
||||
for key, val in cases.items():
|
||||
if not isinstance(val, list):
|
||||
errors.append(
|
||||
f"Switch step {config.get('id', '?')!r}: "
|
||||
f"case {key!r} must be a list of steps."
|
||||
)
|
||||
default = config.get("default")
|
||||
if default is not None and not isinstance(default, list):
|
||||
errors.append(
|
||||
f"Switch step {config.get('id', '?')!r}: "
|
||||
f"'default' must be a list of steps."
|
||||
)
|
||||
return errors
|
||||
68
src/specify_cli/workflows/steps/while_loop/__init__.py
Normal file
68
src/specify_cli/workflows/steps/while_loop/__init__.py
Normal file
@@ -0,0 +1,68 @@
|
||||
"""While loop step — repeat while condition is truthy."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from specify_cli.workflows.base import StepBase, StepContext, StepResult, StepStatus
|
||||
from specify_cli.workflows.expressions import evaluate_condition
|
||||
|
||||
|
||||
class WhileStep(StepBase):
|
||||
"""Repeat nested steps while condition is truthy.
|
||||
|
||||
Evaluates condition *before* each iteration. If falsy on first
|
||||
check, the body never runs. ``max_iterations`` is an optional
|
||||
safety cap (defaults to 10 if omitted).
|
||||
"""
|
||||
|
||||
type_key = "while"
|
||||
|
||||
def execute(self, config: dict[str, Any], context: StepContext) -> StepResult:
|
||||
condition = config.get("condition", False)
|
||||
max_iterations = config.get("max_iterations")
|
||||
if max_iterations is None:
|
||||
max_iterations = 10
|
||||
nested_steps = config.get("steps", [])
|
||||
|
||||
result = evaluate_condition(condition, context)
|
||||
if result:
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={
|
||||
"condition_result": True,
|
||||
"max_iterations": max_iterations,
|
||||
"loop_type": "while",
|
||||
},
|
||||
next_steps=nested_steps,
|
||||
)
|
||||
|
||||
return StepResult(
|
||||
status=StepStatus.COMPLETED,
|
||||
output={
|
||||
"condition_result": False,
|
||||
"max_iterations": max_iterations,
|
||||
"loop_type": "while",
|
||||
},
|
||||
)
|
||||
|
||||
def validate(self, config: dict[str, Any]) -> list[str]:
|
||||
errors = super().validate(config)
|
||||
if "condition" not in config:
|
||||
errors.append(
|
||||
f"While step {config.get('id', '?')!r} is missing "
|
||||
f"'condition' field."
|
||||
)
|
||||
max_iter = config.get("max_iterations")
|
||||
if max_iter is not None:
|
||||
if not isinstance(max_iter, int) or max_iter < 1:
|
||||
errors.append(
|
||||
f"While step {config.get('id', '?')!r}: "
|
||||
f"'max_iterations' must be an integer >= 1."
|
||||
)
|
||||
nested = config.get("steps", [])
|
||||
if not isinstance(nested, list):
|
||||
errors.append(
|
||||
f"While step {config.get('id', '?')!r}: 'steps' must be a list."
|
||||
)
|
||||
return errors
|
||||
Reference in New Issue
Block a user