mirror of
https://github.com/github/spec-kit.git
synced 2026-08-03 06:26:30 +08:00
* feat(workflows): align workflow CLI with extension command surface Adds the missing workflow commands and flags so the workflow CLI matches the extension/preset pattern: add --dev and --from, search --author, update, enable and disable. Disabled workflows are blocked from running and marked in list output. Fixes #2342 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): preserve disabled state on update, guard corrupted registry entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): guard list against corrupted registry entries, re-raise typer.Exit in catalog install workflow list now skips non-dict registry entries with a warning instead of crashing, matching update/enable/disable. The broad except in _install_workflow_from_catalog no longer swallows typer.Exit, so precise errors like the non-HTTPS redirect message are not duplicated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape rich markup in id-mismatch errors and validate --from source early The two id-mismatch error paths interpolated repr() into Rich markup, so a stray bracket in a user typo could be parsed as markup. Route both through rich.markup.escape. `workflow add <source> --from <url>` also validated the source only after downloading. Validate it up front so a URL/path/typo fails without a network fetch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape rich markup in list output and catalog install errors, isolate update failures workflow list now escapes id/name/version/description before printing, matching how extensions render user-editable fields. The catalog install helper computes safe_wf_id once and uses it for every early error path plus the final failure message. workflow update wraps _safe_workflow_id_dir and the backup read inside the try/except typer.Exit block so an unsafe id in a corrupted registry fails that one workflow and the rest continue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape rich markup in --from download exception message Matches how the catalog install path escapes exception strings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): catch OSError in per-workflow update loop and make restore best-effort Transient FS errors (perms, disk full) from backup read or write no longer abort the whole update run. The restore is wrapped in its own try/except so a failed write only warns, and the offending workflow is reported via 'Failed to update' like other per-workflow failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape rich markup in search output workflow search now escapes catalog-derived name/id/version/description/ tags before printing, matching extension search and workflow list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: escape workflow validation errors before Rich output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape remaining unescaped Rich markup paths Covers the last few review threads not yet addressed: - Escape yaml.YAMLError text in the local workflow add install path (matches the already-escaped download/catalog paths). - Escape the non---dev local directory fallback's "No workflow.yml found in <path>" message (the --dev branch already escaped it). - Escape the redirected final_url in the --from non-HTTPS redirect error (IPv6 literals like http://[::1]/... are legal and contain brackets). - Escape the "Downloaded workflow is invalid" exception message in _install_workflow_from_catalog, matching the sibling catalog-install exception handler a few lines above it. Adds regression tests for each in TestWorkflowCliAlignment, following the existing escaping-test pattern in this class. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): escape workflow name/id in install success messages Workflow names and ids come from user-controlled YAML or external catalog data; printing them unescaped lets bracket characters be interpreted as Rich tags. Escape them in the add/catalog-install success messages and the remaining catalog error paths, matching the rest of the output hardening. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): fail cleanly on unparseable catalog install URLs urlparse raises ValueError on e.g. an unbalanced IPv6 literal before the invalid-URL branch is reached; on workflow update that also bypassed the per-workflow handler and aborted the whole command. Convert the parse failure into a clean error so add fails cleanly and update skips just the affected workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): reject catalog updates whose downloaded version mismatches The update path never verified the downloaded workflow carries the catalog version that triggered the update, so a stale or misconfigured URL could report success while leaving the old version installed or downgrading it. Pass the expected version into the install helper and fail the update when the downloaded definition does not match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): validate workflow ID in run command and document new CLI flags Path-equivalent spellings like "align-wf/" previously bypassed the registry disabled check because the engine normalizes the path while the registry matches the raw string. workflow run now validates non-file sources against the workflow ID pattern before lookup. Also updates docs/reference/workflows.md with --dev/--from install options, update/enable/disable commands, and the search --author flag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): enforce disabled state for direct paths to installed workflows Running the installed copy's YAML directly (specify workflow run .specify/workflows/align-wf/workflow.yml) skipped the registry check. File sources resolving inside .specify/workflows/<id>/ now map back to the workflow ID and refuse to run while disabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): reject explicit empty --from URL instead of catalog fallback 'workflow add foo --from ""' fell through 'from_url or ...' to a catalog install. Distinguish None from empty string so explicit values stay on the URL-validation path and fail closed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): registry rollback on save failure, consistent disabled check, honest update summary - WorkflowRegistry.add now rolls back its in-memory mutation when save() raises, so a later successful save cannot persist metadata for a failed update alongside the restored YAML backup. - workflow run uses the same truthiness check for 'enabled' as list and disable, so malformed values like 0 or null refuse to run. - workflow update reports 'No workflows were eligible for update' when every target was skipped instead of claiming all are up to date. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): guard non-string catalog URL and keep enable/disable rollback intact - A truthy non-string catalog url (e.g. 123) reached urlparse and raised AttributeError, escaping the clean error path; validate it is a string. - enable/disable mutated the live registry entry before add(), so add's rollback snapshot captured the already-toggled object; pass a fresh mapping instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): tolerate non-dict registry entries in add and clarify test docstrings A corrupted-but-parseable registry entry (e.g. a string value) crashed WorkflowRegistry.add with AttributeError on existing.get. Guard the non-dict case while still restoring the original raw value on rollback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): atomic registry save and accurate mixed-target update summary - save() wrote the registry with open('w'), so a failed dump truncated the file and the next load reset every entry. Write to a sibling temp file and os.replace into place. - workflow update no longer claims all workflows are up to date when some targets were skipped; it reports checked-only status with a skipped count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): exclusive temp file for registry save and cwd-independent disabled guard - save() now uses tempfile.mkstemp in the workflows dir (matching the engine's atomic writer), so a pre-created symlink at a predictable .tmp path cannot redirect the write and concurrent processes cannot collide. - The direct-path disabled guard derives the owning project from the resolved file path instead of the caller's cwd, so running an installed workflow's YAML from outside the project still refuses when disabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): symlink guards and shape validation in workflow registry, dev-dir file check - WorkflowRegistry now mirrors StepRegistry: _load refuses symlinked parents/registry file and normalizes a non-dict workflows field; save() rejects symlinked paths before writing. - workflow add --dev requires workflow.yml to be a regular file so a directory named workflow.yml gets the documented CLI error instead of an uncaught IsADirectoryError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): validate download redirects before following them All three workflow download sites (add --from, catalog install, step install) passed no redirect_validator to open_url, so an HTTPS URL redirecting to cleartext HTTP issued the insecure request before the post-hoc geturl() check reported it. Shared validator now rejects non-HTTPS redirects (loopback HTTP allowed) pre-follow, matching the preset download path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(workflows): accept redirect_validator kwarg in step-add open_url fakes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): guard directory-shaped workflow.yml and unreadable registry - workflow add's plain local-path fallback (no --dev) checked wf_file.exists() before installing, so a directory literally named workflow.yml passed the guard and _validate_and_install_local() leaked an uncaught IsADirectoryError instead of the documented CLI error. Use is_file(), matching the --dev branch's existing guard. - WorkflowRegistry._load() treated any OSError while reading an existing registry the same as corrupted JSON, resetting to an empty in-memory registry. A later save() would then silently persist that empty state via os.replace, discarding every previously installed workflow entry. Track a _load_error flag on OSError-during-read and have save() refuse to write when it is set, so a transient I/O failure can no longer overwrite intact data on disk. - docs/reference/workflows.md: document `--from <url>` with its value placeholder, matching extensions.md and presets.md. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflow): rollback registry.remove() and guard OSError at CLI boundaries Critical: WorkflowRegistry.remove() deleted the in-memory entry then called save() with no rollback, unlike add(). Combined with workflow_remove deleting the workflow directory before calling registry.remove(), a save failure permanently destroyed the workflow's files, left the on-disk registry still claiming it installed, and surfaced a raw unhandled OSError with no CLI message. - WorkflowRegistry.remove() now rolls back the in-memory entry on a save() OSError, mirroring add()'s existing rollback pattern. - workflow_remove persists the registry removal (registry.remove(), wrapped in try/except OSError -> clean escaped message) before deleting any files, so a save failure never touches the workflow directory. Important sibling paths: workflow add (local/--dev/--from and catalog), enable, and disable all called registry.add() without catching its deliberate OSError, so a save failure surfaced either an orphaned install directory (fresh local/catalog installs) or a raw/unhandled exception with no clean CLI output. - _validate_and_install_local (backs local/--dev/--from) now removes the freshly created directory on a fresh install, or restores the prior workflow.yml bytes on a reinstall-over-existing-local install, before raising a clean escaped error. - _install_workflow_from_catalog wraps the final registry.add() using the function's own established convention (rmtree the just-downloaded workflow_dir, then a clean escaped error) -- workflow_update's existing backup/restore around this function is unaffected. - workflow_enable/workflow_disable catch registry.add()'s OSError and print a clean escaped message instead of leaking the exception. Added failing-first tests proving each behavior (registry-unit rollback test, CLI-level remove/add/enable/disable save-failure tests parametrized where they share one root cause), all confirmed red before the fix and green after. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflow): preserve prior catalog install on reinstall registry-save failure _install_workflow_from_catalog's final registry.add() failure handler unconditionally rmtree'd workflow_dir. That's safe for a brand-new install, but plain `workflow add <catalog-id>` also allows re-adding an already-installed workflow, downloading the new version over the existing directory first. If registry.add() then failed to save, the unconditional rmtree deleted the prior working install while the registry (after its own rollback) still reported it installed -- data loss with no way back. workflow_update already avoids this via an outer backup/restore around this function, but plain add has no such caller. Fix mirrors _validate_and_install_local's existed-before/backup-aware handling: capture whether workflow_dir existed and back up its workflow.yml bytes before any download write, then on a registry.add() OSError, restore those bytes for a reinstall or rmtree only a brand-new directory. Only one file (workflow.yml) is ever written by this path, so no further per-file bookkeeping is needed. Added a failing-first regression: install a catalog workflow, re-add it with a simulated registry save OSError, and assert a clean error, the original workflow.yml restored byte-for-byte, and the registry still reporting the original version installed. Confirmed red (prior file deleted) before the fix, green after. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflow): centralize catalog-install cleanup across all failure branches _install_workflow_from_catalog is new in this PR and has seven failure branches after the mkdir/download step, each independently rmtree'ing workflow_dir: redirect-to-non-HTTPS rejection, a generic download exception, invalid downloaded YAML, a validate_workflow failure, a workflow-id/catalog-key mismatch, a version mismatch, and (fixed in the prior commit) a registry.add() OSError. Only the last one had been special-cased to spare a prior working install on reinstall; the other six still unconditionally deleted the whole directory, so re-adding an already-installed catalog workflow and hitting any of those six earlier failures destroyed the working install even though nothing about it had actually changed. Replaced all seven ad hoc rmtree call sites with a single local _cleanup_failed_install() helper that closes over the existed_before / prior_workflow_bytes captured once at the top of the function: restore the prior workflow.yml for a reinstall, or rmtree only a directory that this attempt itself created. Every failure branch now calls this one helper, so the fix is structural rather than duplicated, and every existing error message/exit code is unchanged -- only the cleanup performed before each message is different. Added a parametrized regression test covering the four early-failure trigger points reachable from plain workflow add (redirect rejection, download exception, invalid YAML, ID mismatch): each installs a catalog workflow, re-adds it while forcing that specific failure, and asserts a clean error plus the original workflow.yml surviving byte-for-byte. Confirmed red against the unfixed code (all four raised FileNotFoundError reading the deleted file) before applying the helper, green after. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflow): restore registry entry verbatim on post-removal rmtree failure workflow_remove now persists registry.remove() before deleting any files (fixed previously), but if the registry write succeeds and the subsequent shutil.rmtree(workflow_dir) then fails, the registry was left claiming the workflow uninstalled while its directory remained on disk -- an orphaned install with no path back to a clean state. workflow_step_remove already handles this exact sequencing by capturing the registry entry before removal and restoring it directly into registry.data plus save() (bypassing add(), which would stamp a new updated_at) if the directory removal fails afterwards. Applied the same pattern to workflow_remove: capture registry_metadata via registry.get() before registry.remove(), and on an rmtree OSError, write it straight back into registry.data["workflows"][workflow_id] and save(), matching workflow_step_remove's restore-failure handling (a yellow warning, not a hard failure, since the primary error is already about to be reported). Existing error message and exit behavior for the rmtree failure are unchanged. Added a failing-first regression: install a workflow, monkeypatch shutil.rmtree to raise OSError, and assert a clean existing error message, the directory remaining (rmtree never actually deleted anything), and the registry entry restored byte-for-byte identical (including installed_at/updated_at) -- proving the fix bypasses add() and doesn't re-stamp timestamps. Confirmed red (registry entry stayed None) before the fix, green after. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 current Copilot review findings on workflow run/registry/install 1. workflow run ownership check followed symlinks via Path.resolve() before mapping a direct YAML path back to its installed workflow ID. A symlinked .specify/workflows/<id>/workflow.yml resolved outside the tree, missed the ownership match entirely, and let the disabled-workflow guard be silently skipped while engine.load_workflow still followed the symlink. Now maps ownership from a lexically-normalized path (os.path. normpath, no symlink following) and explicitly refuses to run if the installed <id> directory or workflow.yml leaf is itself a symlink. Direct external workflow paths that don't match .specify/workflows/... are unaffected. 2. WorkflowRegistry._load() caught a read OSError and silently fell back to an empty in-memory registry, only blocking a later save(). Callers that only query is_installed()/get()/list() before writing a file (e.g. commands/init.py's bundled speckit install, which overwrites workflow.yml once is_installed() reports false) could act on that false-empty state and destroy real data before ever reaching save(). _load() now raises OSError immediately so an unreadable registry fails closed at construction, before any query or side effect is possible. Added _open_workflow_registry() to give every CLI command a consistent clean-error boundary around registry construction. 3. _validate_and_install_local's mkdir/copy2 ran before the try/except that protected registry.add(); a copy2 failure (e.g. a truncating partial write on a reinstall) was not caught at all, so the existing backup-restore cleanup never ran and the prior working workflow.yml was corrupted with a raw traceback surfaced to the user. mkdir/copy2 now run inside the same rollback-protected section as registry.add(), sharing one _cleanup_failed_install() helper. 4. workflow update's skip message claimed any non-catalog source was installed "from a local path or URL", which is wrong for the bundled speckit workflow (source: "bundled"). Message is now source-neutral. Verified all 4 threads are current (not outdated) via GraphQL review thread query on PR #3419, HEAD812050a. Tests: strict TDD per fix (red test proving each bug, minimal production change, green). tests/test_workflows.py: 474 passed. Full suite: 3976 passed, 110 skipped. ruff check: all checks passed on touched files and full src tree. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix disabled-workflow bypass via symlinked .specify project root workflow run's ownership check derived registry_root/registered_id from the lexical path, then checked the id directory and workflow.yml leaf for symlinks -- but never checked .specify or .specify/workflows themselves for that derived root. _reject_unsafe_workflow_storage only guards the cwd's project_root, which can differ from the path-derived registry_root (a direct path into an unrelated project, or that project's own .specify being a symlink to an attacker-controlled tree). WorkflowRegistry's own symlinked-parent handling silently substitutes an empty registry instead of raising, so a query against it (is_installed/get returning "not found") is not a safety signal a caller can rely on: with a symlinked .specify, the disabled check saw no registry entry and let a disabled workflow run anyway. Fix: reject an unsafe .specify/.specify-workflows for the actual derived registry_root before ever consulting the registry, reusing the existing _reject_unsafe_dir helper already used by _reject_unsafe_workflow_storage. Red-first end-to-end repro: victim project's .specify symlinked to an attacker-controlled tree containing a disabled workflow entry, run invoked with a direct path from an unrelated cwd -- confirmed the disabled workflow executed (exit 0) before the fix, now refused cleanly. Tests: tests/test_workflows.py 475 passed. Full suite: 3977 passed, 110 skipped. ruff check: all checks passed. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix raw exception leak in bundle remove primitive boundary remove_bundle() had no exception handling around its component removal loop, unlike install_bundle() which converts any raw exception into a clean BundlerError. Since WorkflowRegistry now fails closed (raises OSError) on an unreadable registry file, and _WorkflowKindManager.__init__ constructs WorkflowRegistry with no try/except, an unreadable workflow registry surfaced as a raw OSError through remove_bundle(). The bundle_remove CLI command only catches BundlerError, so the raw OSError propagated uncaught, producing exit_code=1 with empty output instead of a clean, actionable message. Wrap remove_bundle()'s component loop in the same try/except BundlerError: raise / except Exception: raise BundlerError(...) from exc pattern already used by install_bundle(), converting any raw exception at this shared boundary. save_records() remains outside the try block, so a failure still leaves the bundle's record untouched (no removal side effects recorded). Tests: - tests/integration/test_bundler_install_flow.py::test_remove_converts_raw_installer_exception_to_bundler_error (function-level regression: a raw OSError from installer.is_installed must become a clean BundlerError, and the bundle record must survive) - tests/contract/test_bundle_cli.py::test_remove_reports_clean_error_when_primitive_raises_raw_exception (CLI-level regression: `specify bundle remove` must print a clean actionable message and exit non-zero instead of raw/empty output) Both tests were confirmed red beforehand: the raw OSError propagated uncaught out of remove_bundle(), and the CLI-level CliRunner result showed exit_code=1 with empty output. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 8 current Copilot review findings on registry fail-closed, rollback orphans, backup-read boundaries, and Rich escaping 1. WorkflowRegistry._load(): a symlinked .specify/.specify/workflows parent (or a symlinked registry file) silently returned an empty registry instead of raising, unlike an unreadable-file read failure. A read-only caller (notably the bundler's remove path) querying is_installed() before ever writing could conclude an installed workflow is absent, skip removing it, then delete the bundle record -- leaving the workflow untracked but still on disk. Now raises OSError immediately, matching the existing unreadable-file fail-closed behavior. 2/8. _validate_and_install_local and _install_workflow_from_catalog: when the destination directory already existed but had no prior workflow.yml (e.g. a leftover empty dir), existed_before was True but there were no backup bytes to restore, so the rollback closure did nothing on a later failure -- leaving the newly copied/ downloaded file behind. Both now unlink the newly created file in this case, restoring the pre-existing directory to its prior (empty) state. 3/4. Both install paths read the prior workflow.yml bytes (to seed the reinstall rollback) *before* any try/except boundary: a read failure on the existing file (e.g. a transient permission/FS issue) leaked a raw, unescaped OSError instead of the same clean CLI error used by every other failure branch in these functions. Both reads are now guarded by their own try/except OSError, with no writes attempted before the read succeeds (so there is nothing to roll back on this specific failure). 5. remove_bundle's exception-conversion message unconditionally claimed "No changes were recorded," even though a failure can occur after earlier components in the same bundle have already been removed from disk (save_records never runs on this path, so the record is left claiming the bundle fully installed). The message now reports how many components were already removed when that happened, instead of asserting no changes occurred. 6/7. workflow_remove's new post-registry-removal directory-failure error and its restore-failure warning interpolated workflow_dir and the exception values into Rich markup unescaped. A project path or OS error message containing Rich-markup-like brackets could be parsed as markup and hide/corrupt the displayed text. Both now use the existing _escape_markup helper, consistent with every other error path in this file. Tests (tests/test_workflows.py unless noted): - TestWorkflowRegistry::test_load_symlinked_workflows_dir_fails_closed_not_silently_empty (1) - TestWorkflowCliAlignment::test_add_dev_fresh_install_into_preexisting_empty_dir_cleans_new_file (2) - TestWorkflowCliAlignment::test_add_catalog_fresh_install_into_preexisting_empty_dir_cleans_new_file (8) - TestWorkflowCliAlignment::test_add_dev_reinstall_backup_read_failure_gives_clean_error (3) - TestWorkflowCliAlignment::test_add_catalog_reinstall_backup_read_failure_gives_clean_error (4) - tests/integration/test_bundler_install_flow.py::test_remove_partial_failure_message_reflects_partial_state (5) - TestWorkflowRemoveGuard::test_remove_directory_and_restore_failure_escapes_rich_markup (6/7) All seven were confirmed red beforehand, matching each thread's described failure mode exactly (silent empty registry instead of a raise; orphaned new file left behind; raw unescaped OSError leaking; a misleading "no changes were recorded" claim; Rich markup consuming bracketed path/exception text). Also updated test_registry_save_refuses_symlinked_parent, a pre-existing test that asserted the symlinked-parent raise at add()/save() time -- it now raises at construction instead, per fix #1, so the test was adjusted to match without weakening its guarantee (still asserts no writes occur under the symlinked target). Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 3 current Copilot review findings: bookkeeping-aware BundlerError removal, bounded workflow downloads 1. bundle remove: BundlerError raised by the primitive installer itself (e.g. from a kind manager) bypassed the partial-removal bookkeeping message added previously via a bare `except BundlerError: raise`. Now routes through the same detail-construction logic as generic exceptions, so a mid-loop BundlerError after an earlier successful removal still reports that the project may be partially uninstalled, while a zero-removal BundlerError still reports "No components were removed." Both preserve the original exception message and chain `from exc`. 2/3. workflow add --from and catalog install/update downloads used unbounded `response.read()`, buffering the entire server-controlled body into memory before any size check, and trusted Content-Length alone where checked at all. Added a single shared `_read_response_within_limit()` helper reused by both call sites: it fails fast on an oversized declared Content-Length, and separately enforces the same cap while streaming in 64KiB chunks so a chunked or Content-Length-less response cannot bypass the limit by lying about or omitting its size. Chose 5 MiB as the cap: workflow YAML definitions are small step/metadata text, not binaries, so this is generous headroom against a malicious/misbehaving server without affecting any legitimate workflow definition. Both call sites already route any raised exception through their existing clean-error and rollback (`_cleanup_failed_install`) paths, so no additional error-handling plumbing was needed. Tests: extended the shared `_FakeResponse` test helper (and 5 duplicate per-test FakeResponse classes) to support `.read(amt)` chunked reads with an internal cursor (backward compatible with existing bare `.read()` callers) plus header simulation. Added red-first tests for: BundlerError after partial removal reporting partial state, BundlerError with zero removals reporting no changes, --from oversized-Content-Length rejection, --from oversized-streamed-body-without-Content-Length rejection, and the same two cases for the catalog install path (asserting no orphan directory/registry mutation on rejection). tests/integration/test_bundler_install_flow.py: 17 passed tests/test_workflows.py: 485 passed tests -q: 3992 passed, 110 skipped ruff check: clean on all touched files Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix temp-file leak in workflow add --from and strengthen size-limit test assertions workflow_add's --from download path opened a NamedTemporaryFile(delete=False) -- which creates the file on disk immediately -- then wrote the size-limited response body before assigning `tmp_path`. If `_read_response_within_limit` raised (oversized declared Content-Length, or an over-cap streamed body with no/understated Content-Length), the exception propagated out of the `with` block before `tmp_path` was ever set, so the outer except handler had no path to clean up: a 0-byte `.yml` temp file was left behind permanently on every rejected/failed --from download. Fixed by assigning `tmp_path` immediately after the file is opened (before the size-limited read/write), and unlinking it in the except branch when set. Normal post-download cleanup in the existing `finally: tmp_path.unlink(missing_ok=True)` is unchanged. Verified (not assumed) the catalog install path has no equivalent leak: it writes the response bytes directly to `workflow_file` inside `workflow_dir` (no separate temp file), and any read/size-limit failure is already caught by the existing `except Exception: _cleanup_failed_install()` handler, which correctly restores a reinstalled file or removes a freshly-created directory. While investigating, found the previous round's 4 size-limit tests were false positives: `_read_response_within_limit`'s `max_bytes` parameter had its default bound to `_MAX_WORKFLOW_YAML_BYTES` at function-definition time, so monkeypatching the module attribute in tests had no effect on the function's actual behavior -- the tests were passing because the oversized mock bodies failed downstream YAML/id validation instead of the size check. Fixed by resolving `max_bytes` from the module attribute at call time (default `None`, resolved inside the function body) so tests can actually override the effective limit, and strengthened all 4 tests' assertions to match the specific size-limit error text (whitespace-collapsed to tolerate Rich's line-wrapping), so they now prove the real code path fires. Tests: added 2 red-first regression tests (oversized-streamed-body and oversized-Content-Length --from downloads leave no leftover temp file, verified against a scratch tempfile.tempdir), confirmed red (real 0-byte file found) before the fix and green after. Strengthened the pre-existing 4 --from/catalog size-limit tests to assert on the actual error message instead of generic exit-code/non-empty-output checks. tests/test_workflows.py: 487 passed tests -k bundler: 186 passed tests -q: 3994 passed, 110 skipped ruff check: clean on all touched files Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden workflow install/remove transactions with atomic staging Addresses 5 Copilot review findings on HEADb8269c8, all centered on transaction integrity around workflow install/remove/registry writes, following the atomic_write_json pattern already used in _utils.py: 1. WorkflowRegistry.save() now preserves the existing registry file's mode (e.g. 0640/0644) across a save instead of silently downgrading it to mkstemp's 0600 default; a brand-new registry still gets the secure 0600 default. 2. workflow_remove now stages the install directory out of the way via an atomic rename *before* the registry write, rather than deleting it directly with shutil.rmtree after the registry already claims it removed. This closes a real data-integrity gap: a partially-failed rmtree could no longer leave a damaged directory re-marked "installed" by the old manual restore-after-rmtree-failure code (now deleted -- it's structurally impossible to need it). A registry-write failure renames the staged directory back (guarded, with an explicit warning if the restore-back rename itself fails); a registry-write success is durable, so a later failure to delete the staged directory is now a warning (exit 0), not a contradictory "Error: Failed to remove" (exit 1) that used to claim failure while the registry already recorded success. 3. Local (--dev/--from/plain path) and catalog install/reinstall now write new content to a same-directory staging file and commit it onto the destination workflow.yml via a single atomic swap, instead of writing/downloading directly into the destination file. A prior file (reinstall) is renamed aside rather than overwritten in place, so it can be restored via rename -- never a content rewrite -- if registry.add() subsequently fails; a rollback failure is now explicitly reported as a warning instead of escaping unguarded and masking the original clean error. This also removes the need to read the prior file's bytes into memory before installing (that read-before-write step and its failure mode are now unreachable), and both local and catalog installs share the same four small helpers (_stage_workflow_file / _commit_workflow_file / _discard_staged_workflow_file / _rollback_committed_workflow_file, plus guarded wrappers) rather than duplicating the logic. 4. Updated a stale comment (workflow_run's ownership-guard rationale) that still described WorkflowRegistry._load() as silently substituting an empty registry; it now fails closed by raising OSError, which the comment now states plainly. Tests: rewrote the two workflow_remove tests whose assertions encoded the old (incoherent) rmtree-then-restore contract to instead prove the new stage-then-commit contract (post-registry-success cleanup failure is a warning+exit 0; pre-registry-success stage-restore failure is guarded and escapes markup correctly). Rewrote the local/catalog "backup read failure" tests, which tested a step the new design no longer performs, into "restore-rename failure" tests proving the new guarded rollback boundary. Added registry file-mode preservation tests. All other existing install/remove/reinstall tests (save-failure cleanup, pre-existing-empty-dir handling, early-failure-during- reinstall parametrized cases, Rich markup escaping) continue to pass unmodified against the new implementation. Verified via GraphQL that all 5 threads are current (not outdated/ resolved) before fixing. Full suite: 3996 passed, 110 skipped. Ruff clean on all touched files. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Discard reinstall backup file after registry.add() succeeds _commit_workflow_file() renames a prior workflow.yml aside to workflow.yml.bak so it can be restored if registry.add() subsequently fails. Neither the local install/reinstall path nor the catalog install/reinstall path ever cleaned up that backup after a successful registry.add() -- every successful reinstall permanently left a workflow.yml.bak sibling, which later reinstalls would silently overwrite/re-orphan. Add a shared _discard_committed_backup_file() helper, called from both success paths right after registry.add() durably succeeds (and before the final "installed" message, preserving output ordering). A fresh install (backup_file is None) is a no-op. A cleanup failure is reported as a warning (exit 0), not a failure, since the install itself already succeeded -- consistent with workflow_remove's post-commit cleanup warning semantics. Add red-first regression tests proving: (1) successful local reinstall leaves no workflow.yml.bak sibling, (2) successful catalog reinstall leaves no workflow.yml.bak sibling, (3) a cleanup failure on the backup file after a successful reinstall reports a warning and still exits 0 with the registry correctly reflecting the new install. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up freshly-created dest_dir when staging mkstemp fails _stage_workflow_file() does dest_dir.mkdir(parents=True, exist_ok=True) then tempfile.mkstemp(dir=dest_dir, ...). For a fresh install (no prior directory), if mkdir succeeds but mkstemp then raises (disk full/EMFILE/quota), the exception previously propagated straight past both the local-install and catalog-install call sites without any cleanup, leaving the newly-created empty workflow directory orphaned on disk with no error indicating why. Fix at the shared _stage_workflow_file() boundary instead of duplicating cleanup at each call site: track whether this call created dest_dir: on a mkstemp failure, remove that directory via a guarded rmdir (never a broad rmtree, so any concurrently written content would be left untouched) before re-raising the original OSError unchanged. A pre-existing (reinstall) dest_dir is never touched by this cleanup, and a cleanup failure is reported as its own warning without masking the original error. Add red-first regression tests proving: a fresh local install (--dev, plain local path, --from) and a fresh catalog install both clean up the orphaned directory on a simulated mkstemp failure, and a reinstall over a pre-existing directory is left untouched by the same failure. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix installed-workflow ownership/disabled bypass and resume enforcement Address 3 current Copilot review findings on the disabled-workflow guard in `workflow run`/`workflow resume`: - The lexical `.specify/workflows/<id>` ownership scan stopped at the first match scanning from the start of the path. A nested project living beneath an outer installed workflow's own directory tree (reusing the same segment names) was attributed to the wrong (outer) workflow and ID, gating the run on an unrelated workflow's disabled state. `_scan_for_workflow_owner` now scans from the end so the nearest (innermost) owner always wins. - A path with no `.specify/workflows` segments of its own (e.g. `/tmp/alias.yml`) that is itself a symlink resolving *into* installed storage bypassed the disabled check entirely, since only the raw lexical path was inspected. `_resolve_installed_workflow_ownership` now additionally resolves the real path when the lexical scan finds no owner and re-runs the same scan against it, so an outward-pointing alias into a disabled workflow is caught too. Genuinely standalone external files (no symlink anywhere on the path) are unaffected. - `workflow resume` bypassed the disabled check altogether: engine.resume() replays a persisted run directly from disk with no registry awareness. RunState now optionally persists `installed_workflow_id` and `installed_registry_root` at run start (set by workflow_run when the source resolved to an installed ID); `workflow_resume` pre-loads the run state and re-checks the registry's *current* disabled state before calling engine.resume(), mirroring workflow_run's own guard. Both new fields default to None via RunState.load()'s `.get()`, so runs from a direct/non-installed source, and any run persisted before this schema addition, resume exactly as before. The ownership-mapping logic (previously inlined in workflow_run) is extracted into `_resolve_installed_workflow_ownership` / `_scan_for_workflow_owner` so both the lexical and resolved-path cases share the same scan and the existing inward-symlink-component refusal. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Guard --from temp cleanup; drop redundant update rollback; mark POSIX-only tests Two more current Copilot review findings, both in workflow_add/update: - `workflow add --from`'s `finally: tmp_path.unlink(missing_ok=True)` ran unguarded after `_validate_and_install_local` had already committed the file and registry entry (success) or already raised its own clean `typer.Exit` (failure). An OSError from that cleanup unlink would surface as an unhandled failure even though the install itself succeeded. It is now wrapped in try/except OSError, printing a neutral warning that doesn't claim success or failure (the finally runs on both outcomes) instead of propagating. - `workflow_update`'s per-item loop performed its own outer backup (`wf_file.read_bytes()`) and restore (`wf_file.write_bytes(backup)`) around `_install_workflow_from_catalog`, which is itself fully transactional (staged download, atomic rename-based commit, its own rollback on registry failure) and never leaves a raw OSError or a partially-written workflow.yml. The outer restore was therefore dead weight for its stated purpose, and — being an unguarded byte-level write — was itself an unnecessary place a second failure could truncate an already-safely-preserved file. Removed; the loop now only records success/failure. Also marks 3 registry-save file-mode tests (`test_registry_save_preserves_existing_file_mode`, `test_registry_save_on_new_registry_uses_secure_default_mode`, `test_registry_save_failure_preserves_file_on_disk`) as POSIX-only via the repo's existing `skipif(sys.platform == "win32", ...)` pattern, since they assert exact POSIX permission bits that don't hold on Windows. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Report possible partial changes on zero-removed bundle removal failure The final Copilot review finding: `remove_bundle`'s zero-removed-components error message claimed "No components were removed." even when the failing installer component may have deleted files before raising -- prior review rounds already established that DefaultPrimitiveInstaller's removal paths are not atomic and can leave partial filesystem changes despite raising before `result.uninstalled` is populated. The zero-count message is now a conservative caution ("...but the failing component may have made partial changes before raising, so the project may be partially uninstalled.") instead of an unconditional claim of no side effects. The >0-removed path (which already reports the confirmed partial list) is unchanged. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix workflow resume disabled-check bypass after project move/rename RunState.installed_registry_root previously persisted the creation-time absolute project path unconditionally whenever a run belonged to an installed workflow. After the whole project directory was renamed or moved, workflow_resume would open a WorkflowRegistry at that now nonexistent path, get back an empty/default registry, and silently skip the disabled-workflow check -- a paused run for a disabled workflow could be resumed successfully from the new location. Fix persists installed_registry_root only when the owning root genuinely differs from the current project_root (true cross-project direct-file- source invocations). The common same-project case now persists None and is re-derived from the live project_root at resume time via a new _resolve_run_owner_root() helper, which also falls back to project_root if a stored root no longer exists on disk -- covering both the common case transparently surviving project moves and the cross-project case degrading safely if its owner project vanishes, rather than silently skipping the disabled check. Backward compatible: state files missing the new fields, and states with a still-existing distinct cross-project root, behave unchanged. Added regression tests: - resume blocked after project moved then disabled at new location - resume still works after project moved while workflow stays enabled - cross-project registry root is still correctly honored when it exists - resume falls back to current project's registry when a stored cross-project root no longer exists Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix silenced cleanup failures and malformed run-state type validation Four fixes from Copilot review on HEAD4f24735: 1. _discard_staged_workflow_file's fresh-install directory removal used shutil.rmtree(dest_dir, ignore_errors=True), so a genuine cleanup failure there could never reach _safe_discard_staged_workflow_file's warning -- an orphaned directory was left behind with zero report. Now removes only if dest_dir still exists and lets a real OSError propagate to the existing safe wrapper, which warns while the already-printed original install error remains primary. 2. _rollback_committed_workflow_file's fresh-install directory removal (post registry.add() failure) had the same ignore_errors=True gap; fixed identically so _safe_rollback_committed_workflow_file's warning can actually fire. 3. In the --from download-failure branch, tmp_path.unlink(missing_ok= True) was unguarded: if it raised (e.g. read-only tempdir), it replaced the original "Failed to download workflow" error with a raw unhandled OSError instead of a clean typer.Exit. Now guarded exactly like the later post-install finally cleanup: a cleanup failure prints a warning and the original download error is still reported cleanly. 4. RunState.load() trusted installed_workflow_id/installed_registry_root straight out of state.json with no type validation. A malformed value (int/list/dict/bool instead of str-or-null) would crash deep inside _resolve_run_owner_root or the registry lookup (TypeError building a Path, unhashable dict/list as a mapping key) instead of failing cleanly. Both fields are now validated as str | None during load, raising a clear ValueError that workflow_resume's existing ValueError boundary already converts into a clean CLI error with no traceback. Valid values (including the empty-string fallback already handled by _resolve_run_owner_root) continue to load unchanged. Added red-first regression tests for each: staged-discard cleanup warning, rollback cleanup warning, download-failure cleanup-vs-original- error precedence, and parameterized malformed/valid run-state field coverage. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add ValueError boundary to workflow status single-run lookup QUALITY re-review flagged that fa410d3's new RunState.load() type validation (malformed installed_workflow_id/installed_registry_root raising ValueError) leaked as a raw unhandled traceback through `workflow status <run_id>`, which only caught FileNotFoundError. `workflow resume` already had the matching ValueError boundary. Adds an `except ValueError as exc: console.print(f"[red]Error:[/red] {exc}"); raise typer.Exit(1)` clause mirroring resume's exact pattern (unescaped interpolation, consistent with the existing convention at every other ValueError boundary in this file). FileNotFoundError behavior and the no-run-id list-all-runs path are unchanged. Added parametrized regression covering malformed installed_workflow_id/ installed_registry_root (int/list) via `workflow status`, plus regressions locking in the unaffected FileNotFoundError and no-run-id list-path behaviors. Assisted-by: GitHub Copilot (model: Claude Sonnet 5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: bound workflow step downloads Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: preserve workflow reinstall state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: fail closed on workflow registry state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make workflow installs transactional Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: close workflow transaction races Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: clean up failed workflow transactions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: clean up workflow removal state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: guard workflow update transactions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: harden workflow lifecycle edge cases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: verify installed workflow ownership Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: isolate workflow rollback cleanup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: preserve unique workflow backups Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: bind workflow staging to file descriptors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: fail closed on corrupt workflow registry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: bind workflow ownership and source identity Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): harden redirects and Windows tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): restore staged removals on serialization errors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): preserve state across interrupted writes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): harden resume ownership checks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): validate persisted run state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(workflows): validate origin and release metadata Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1491 lines
65 KiB
Python
1491 lines
65 KiB
Python
"""Workflow engine — loads, validates, and executes workflow YAML definitions.
|
|
|
|
The engine is the orchestrator that:
|
|
- Parses workflow YAML definitions
|
|
- Validates step configurations and requirements
|
|
- Executes steps sequentially, dispatching to the correct step type
|
|
- Manages state persistence for resume capability
|
|
- Handles control flow (branching, loops, fan-out/fan-in)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import dataclasses
|
|
import json
|
|
import os
|
|
import re
|
|
import tempfile
|
|
import threading
|
|
import uuid
|
|
from concurrent.futures import Future, ThreadPoolExecutor
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import yaml
|
|
|
|
from ..integration_state import (
|
|
default_integration_key,
|
|
try_read_integration_json,
|
|
)
|
|
from .base import RunStatus, StepContext, StepResult, StepStatus
|
|
|
|
|
|
# -- Workflow Definition --------------------------------------------------
|
|
|
|
|
|
class WorkflowDefinition:
|
|
"""Parsed and validated workflow YAML definition."""
|
|
|
|
def __init__(self, data: dict[str, Any], source_path: Path | None = None) -> None:
|
|
self.data = data
|
|
self.source_path = source_path
|
|
|
|
workflow = data.get("workflow", {})
|
|
self.id: str = workflow.get("id", "")
|
|
self.name: str = workflow.get("name", "")
|
|
self.version: str = workflow.get("version", "0.0.0")
|
|
self.author: str = workflow.get("author", "")
|
|
self.description: str = workflow.get("description", "")
|
|
self.schema_version: str = data.get("schema_version", "1.0")
|
|
|
|
# Defaults
|
|
self.default_integration: str | None = workflow.get("integration")
|
|
self.default_model: str | None = workflow.get("model")
|
|
self.default_options: dict[str, Any] = workflow.get("options") or {}
|
|
if not isinstance(self.default_options, dict):
|
|
self.default_options = {}
|
|
|
|
# Advisory pre-conditions (spec-kit version / integrations a workflow
|
|
# expects). Validated by ``validate_workflow`` (recognized keys only;
|
|
# see ``_RECOGNIZED_REQUIRES_KEYS``) but NOT enforced at run time — they
|
|
# are not a security boundary. In particular there is no
|
|
# ``requires.permissions`` capability gate: shell steps always run with
|
|
# the user's privileges.
|
|
#
|
|
# Holds the raw parsed value, so before ``validate_workflow`` runs it may
|
|
# be a non-mapping (``None`` for a bare ``requires:``, a list for
|
|
# ``requires: []``, etc.); typed ``Any`` rather than ``dict[str, Any]``
|
|
# to avoid implying it is always a mapping at this point.
|
|
self.requires: Any = data.get("requires", {})
|
|
|
|
# Inputs
|
|
self.inputs: dict[str, Any] = data.get("inputs", {})
|
|
|
|
# Steps
|
|
self.steps: list[dict[str, Any]] = data.get("steps", [])
|
|
|
|
@classmethod
|
|
def from_yaml(cls, path: Path) -> WorkflowDefinition:
|
|
"""Load a workflow definition from a YAML file."""
|
|
with open(path, encoding="utf-8") as f:
|
|
data = yaml.safe_load(f)
|
|
if not isinstance(data, dict):
|
|
msg = f"Workflow YAML must be a mapping, got {type(data).__name__}."
|
|
raise ValueError(msg)
|
|
return cls(data, source_path=path)
|
|
|
|
@classmethod
|
|
def from_string(cls, content: str) -> WorkflowDefinition:
|
|
"""Load a workflow definition from a YAML string."""
|
|
data = yaml.safe_load(content)
|
|
if not isinstance(data, dict):
|
|
msg = f"Workflow YAML must be a mapping, got {type(data).__name__}."
|
|
raise ValueError(msg)
|
|
return cls(data)
|
|
|
|
|
|
# -- Workflow Validation --------------------------------------------------
|
|
|
|
# ID format: lowercase alphanumeric with hyphens
|
|
_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$")
|
|
|
|
# Keys accepted under a workflow's ``requires`` block: the advisory
|
|
# pre-conditions documented for workflows (``speckit_version`` and
|
|
# ``integrations``). This is the *workflow* schema only — the bundle manifest's
|
|
# ``requires`` (see ``bundler/models/manifest.py``) is a separate schema that
|
|
# also carries ``tools``/``mcp``; those are not workflow ``requires`` keys.
|
|
# Any other key — notably ``permissions`` — is rejected by ``validate_workflow``
|
|
# so it is never mistaken for an enforced runtime control.
|
|
_RECOGNIZED_REQUIRES_KEYS = frozenset({"speckit_version", "integrations"})
|
|
|
|
# Valid step types (matching STEP_REGISTRY keys)
|
|
def _get_valid_step_types() -> set[str]:
|
|
"""Return valid step types from the registry, with a built-in fallback."""
|
|
from . import STEP_REGISTRY
|
|
if STEP_REGISTRY:
|
|
return set(STEP_REGISTRY.keys())
|
|
return {
|
|
"command", "shell", "prompt", "gate", "if", "init",
|
|
"switch", "while", "do-while", "fan-out", "fan-in",
|
|
}
|
|
|
|
|
|
def validate_workflow(definition: WorkflowDefinition) -> list[str]:
|
|
"""Validate a workflow definition and return a list of error messages.
|
|
|
|
An empty list means the workflow is valid.
|
|
"""
|
|
errors: list[str] = []
|
|
|
|
# -- Schema version ---------------------------------------------------
|
|
# str() so an unquoted ``schema_version: 1.0`` (YAML float) is accepted —
|
|
# rejecting it would print "Unsupported schema_version 1.0. Expected '1.0'."
|
|
if str(definition.schema_version) != "1.0":
|
|
errors.append(
|
|
f"Unsupported schema_version {definition.schema_version!r}. "
|
|
f"Expected '1.0'."
|
|
)
|
|
|
|
# -- Top-level fields -------------------------------------------------
|
|
# YAML parses unquoted scalars like ``id: 123`` or ``version: 1.0`` as
|
|
# int/float; check types before regex/string operations so authoring
|
|
# mistakes surface as validation errors instead of tracebacks. Only
|
|
# ``None``/empty-string count as missing so falsey non-strings
|
|
# (``id: 0``, ``name: false``) still get the typed error.
|
|
if definition.id is None or definition.id == "":
|
|
errors.append("Workflow is missing 'workflow.id'.")
|
|
elif not isinstance(definition.id, str):
|
|
errors.append(
|
|
f"'workflow.id' must be a string, got "
|
|
f"{type(definition.id).__name__} ({definition.id!r})."
|
|
)
|
|
elif not _ID_PATTERN.fullmatch(definition.id):
|
|
errors.append(
|
|
f"Workflow ID {definition.id!r} must be lowercase alphanumeric "
|
|
f"with hyphens."
|
|
)
|
|
|
|
if definition.name is None or definition.name == "":
|
|
errors.append("Workflow is missing 'workflow.name'.")
|
|
elif not isinstance(definition.name, str):
|
|
errors.append(
|
|
f"'workflow.name' must be a string, got "
|
|
f"{type(definition.name).__name__} ({definition.name!r})."
|
|
)
|
|
|
|
if definition.version is None or definition.version == "":
|
|
errors.append("Workflow is missing 'workflow.version'.")
|
|
elif not isinstance(definition.version, str):
|
|
errors.append(
|
|
f"'workflow.version' must be a string, got "
|
|
f"{type(definition.version).__name__} ({definition.version!r}) — "
|
|
f'quote it in YAML (version: "1.0.0").'
|
|
)
|
|
elif not re.fullmatch(r"\d+\.\d+\.\d+", definition.version):
|
|
errors.append(
|
|
f"Workflow version {definition.version!r} is not valid "
|
|
f"semantic versioning (expected X.Y.Z)."
|
|
)
|
|
|
|
# -- Inputs -----------------------------------------------------------
|
|
if not isinstance(definition.inputs, dict):
|
|
errors.append("'inputs' must be a mapping (or omitted).")
|
|
else:
|
|
for input_name, input_def in definition.inputs.items():
|
|
if not isinstance(input_def, dict):
|
|
errors.append(f"Input {input_name!r} must be a mapping.")
|
|
continue
|
|
input_type = input_def.get("type")
|
|
if input_type and input_type not in ("string", "number", "boolean"):
|
|
errors.append(
|
|
f"Input {input_name!r} has invalid type {input_type!r}. "
|
|
f"Must be 'string', 'number', or 'boolean'."
|
|
)
|
|
|
|
# Validate the default eagerly so authoring mistakes (e.g. a
|
|
# default not in the declared enum, or a non-numeric default for
|
|
# a number input) surface at install/validation time instead of
|
|
# at workflow-execution time. ``"auto"`` for the integration
|
|
# input is a runtime-resolved sentinel, so only the
|
|
# enum-membership check is exempted for that exact case — the
|
|
# declared type is still enforced (e.g. ``type: number`` paired
|
|
# with ``default: "auto"`` is still rejected).
|
|
if "default" in input_def:
|
|
default_value = input_def["default"]
|
|
is_auto_integration = (
|
|
input_name == "integration" and default_value == "auto"
|
|
)
|
|
validation_input_def: dict[str, Any] = input_def
|
|
if is_auto_integration and "enum" in input_def:
|
|
validation_input_def = {
|
|
key: value
|
|
for key, value in input_def.items()
|
|
if key != "enum"
|
|
}
|
|
try:
|
|
WorkflowEngine._coerce_input(
|
|
input_name, default_value, validation_input_def
|
|
)
|
|
except ValueError as exc:
|
|
errors.append(
|
|
f"Input {input_name!r} has invalid default: {exc}"
|
|
)
|
|
|
|
# -- Requires ---------------------------------------------------------
|
|
# ``requires`` declares advisory pre-conditions (the spec-kit version and
|
|
# integrations a workflow expects). Only a fixed set of keys is recognized;
|
|
# reject anything else so authoring typos surface here instead of being
|
|
# silently ignored at runtime. In particular ``requires.permissions`` is
|
|
# rejected explicitly: it reads like a runtime capability gate, but no such
|
|
# gate exists — a ``shell`` step always runs with the user's privileges, so
|
|
# declaring it would give a false sense of sandboxing.
|
|
#
|
|
# Mirror ``inputs`` validation: an omitted block defaults to ``{}`` and is
|
|
# valid, but any present-but-non-mapping value — ``requires:`` (YAML null),
|
|
# ``requires: []`` or ``requires: ''`` — is an authoring error and must
|
|
# surface here rather than be silently ignored at runtime.
|
|
if not isinstance(definition.requires, dict):
|
|
errors.append("'requires' must be a mapping (or omitted).")
|
|
else:
|
|
for key in definition.requires:
|
|
if key == "permissions":
|
|
errors.append(
|
|
"'requires.permissions' is not a recognized or "
|
|
"enforced capability gate — shell steps always run "
|
|
"with the user's privileges. Remove it and gate "
|
|
"sensitive steps with a 'gate' step instead."
|
|
)
|
|
elif key not in _RECOGNIZED_REQUIRES_KEYS:
|
|
errors.append(
|
|
f"Unknown 'requires' key {key!r}. Recognized keys: "
|
|
f"{', '.join(sorted(_RECOGNIZED_REQUIRES_KEYS))}."
|
|
)
|
|
|
|
# -- Steps ------------------------------------------------------------
|
|
if not isinstance(definition.steps, list):
|
|
errors.append("'steps' must be a list.")
|
|
return errors
|
|
if not definition.steps:
|
|
errors.append("Workflow has no steps defined.")
|
|
|
|
seen_ids: set[str] = set()
|
|
_validate_steps(definition.steps, seen_ids, errors)
|
|
|
|
return errors
|
|
|
|
|
|
def _validate_steps(
|
|
steps: list[dict[str, Any]],
|
|
seen_ids: set[str],
|
|
errors: list[str],
|
|
) -> None:
|
|
"""Recursively validate a list of steps."""
|
|
from . import STEP_REGISTRY
|
|
|
|
for step_config in steps:
|
|
if not isinstance(step_config, dict):
|
|
errors.append(f"Step must be a mapping, got {type(step_config).__name__}.")
|
|
continue
|
|
|
|
step_id = step_config.get("id")
|
|
if step_id is None or step_id == "":
|
|
errors.append("Step is missing 'id' field.")
|
|
continue
|
|
if not isinstance(step_id, str):
|
|
errors.append(
|
|
f"Step ID must be a string, got "
|
|
f"{type(step_id).__name__} ({step_id!r})."
|
|
)
|
|
continue
|
|
|
|
if ":" in step_id:
|
|
errors.append(
|
|
f"Step ID {step_id!r} contains ':' which is reserved "
|
|
f"for engine-generated nested IDs (parentId:childId)."
|
|
)
|
|
|
|
if step_id in seen_ids:
|
|
errors.append(f"Duplicate step ID {step_id!r}.")
|
|
seen_ids.add(step_id)
|
|
|
|
# Determine step type
|
|
step_type = step_config.get("type", "command")
|
|
if step_type not in _get_valid_step_types():
|
|
errors.append(
|
|
f"Step {step_id!r} has invalid type {step_type!r}."
|
|
)
|
|
continue
|
|
|
|
# Delegate to step-specific validation
|
|
step_impl = STEP_REGISTRY.get(step_type)
|
|
if step_impl:
|
|
step_errors = step_impl.validate(step_config)
|
|
errors.extend(step_errors)
|
|
|
|
# Validate optional `continue_on_error` field. The engine honours
|
|
# this on any step that returns StepStatus.FAILED so the pipeline can route
|
|
# around the failure via a downstream `if` or `switch` (or a
|
|
# `gate` that surfaces the failure to the operator via message
|
|
# interpolation). The field must be a literal boolean —
|
|
# coercion from truthy strings is deliberately not supported so
|
|
# authoring mistakes surface at validation time rather than
|
|
# silently changing run semantics.
|
|
if "continue_on_error" in step_config:
|
|
coe = step_config["continue_on_error"]
|
|
if not isinstance(coe, bool):
|
|
errors.append(
|
|
f"Step {step_id!r}: 'continue_on_error' must be a "
|
|
f"boolean, got {type(coe).__name__}."
|
|
)
|
|
|
|
# Fan-in: every wait_for id must reference a step declared at or before
|
|
# this point. An id not yet seen is either a typo (unknown step) or a
|
|
# forward reference (the target runs after this fan-in, so its results
|
|
# cannot exist yet) — both are wiring errors that previously surfaced as
|
|
# a silent empty result + COMPLETED. A step that is declared but only
|
|
# conditionally executed (e.g. inside an if/switch branch) is still
|
|
# "seen" here, so a legitimately-empty result at runtime stays valid.
|
|
if step_type == "fan-in":
|
|
wait_for = step_config.get("wait_for")
|
|
if isinstance(wait_for, list):
|
|
for wid in wait_for:
|
|
if not isinstance(wid, str):
|
|
# A non-string entry (e.g. YAML `wait_for: [123]`) can
|
|
# never match a real step id, so the join is silently
|
|
# empty at runtime — surface it as a wiring error.
|
|
errors.append(
|
|
f"Fan-in step {step_id!r}: 'wait_for' entries must "
|
|
f"be step-id strings, got {type(wid).__name__} "
|
|
f"({wid!r})."
|
|
)
|
|
elif wid == step_id:
|
|
# The fan-in's own id is already in seen_ids by now, so
|
|
# a self-reference would pass the membership check below
|
|
# while still producing an empty join at runtime.
|
|
errors.append(
|
|
f"Fan-in step {step_id!r}: 'wait_for' references "
|
|
f"itself; a fan-in cannot wait for its own results."
|
|
)
|
|
elif wid not in seen_ids:
|
|
errors.append(
|
|
f"Fan-in step {step_id!r}: 'wait_for' references "
|
|
f"unknown or not-yet-declared step id {wid!r}."
|
|
)
|
|
|
|
# Recursively validate nested steps
|
|
for nested_key in ("then", "else", "steps"):
|
|
nested = step_config.get(nested_key)
|
|
if isinstance(nested, list):
|
|
_validate_steps(nested, seen_ids, errors)
|
|
|
|
# Validate switch cases
|
|
cases = step_config.get("cases")
|
|
if isinstance(cases, dict):
|
|
for _case_key, case_steps in cases.items():
|
|
if isinstance(case_steps, list):
|
|
_validate_steps(case_steps, seen_ids, errors)
|
|
|
|
# Validate switch default
|
|
default = step_config.get("default")
|
|
if isinstance(default, list):
|
|
_validate_steps(default, seen_ids, errors)
|
|
|
|
# Validate fan-out nested step (template — not added to seen_ids
|
|
# since the engine generates parentId:templateId:index at runtime)
|
|
fan_step = step_config.get("step")
|
|
if isinstance(fan_step, dict):
|
|
fan_errors: list[str] = []
|
|
_validate_steps([fan_step], set(), fan_errors)
|
|
errors.extend(fan_errors)
|
|
|
|
|
|
# -- Run State Persistence ------------------------------------------------
|
|
|
|
|
|
class RunState:
|
|
"""Manages workflow run state for persistence and resume."""
|
|
|
|
# ``run_id`` is interpolated into a filesystem path (``runs/<run_id>``)
|
|
# by both ``save()`` and ``load()``. Constrain it to a charset that
|
|
# cannot contain path separators (``/`` ``\``), parent-directory
|
|
# segments (``..``), or NULs — anything that could escape the
|
|
# ``.specify/workflows/runs/`` directory or be mis-interpreted by the
|
|
# filesystem. The first-character anchor blocks IDs that start with
|
|
# ``-`` (which would be mistaken for a CLI flag in error messages
|
|
# and shell completions).
|
|
_RUN_ID_PATTERN = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_-]*$")
|
|
|
|
@classmethod
|
|
def _validate_run_id(cls, run_id: str) -> None:
|
|
"""Raise ``ValueError`` if ``run_id`` is not a safe path component.
|
|
|
|
This is the single source of truth for what counts as a valid
|
|
``run_id``. ``__init__`` calls it to reject malformed IDs at
|
|
construction time; ``load`` calls it *before* interpolating the
|
|
ID into a path so a malicious value cannot probe or read files
|
|
outside ``.specify/workflows/runs/<run_id>/``.
|
|
"""
|
|
if not isinstance(run_id, str) or not cls._RUN_ID_PATTERN.fullmatch(run_id):
|
|
raise ValueError(
|
|
f"Invalid run_id {run_id!r}: must be alphanumeric with "
|
|
"hyphens/underscores only (and must start with an "
|
|
"alphanumeric character)."
|
|
)
|
|
|
|
@staticmethod
|
|
def _validate_installed_origin(
|
|
installed_workflow_id: str | None,
|
|
installed_registry_root: str | None,
|
|
) -> None:
|
|
"""Validate persisted installed-workflow ownership metadata."""
|
|
if installed_workflow_id is not None:
|
|
if not isinstance(installed_workflow_id, str):
|
|
raise ValueError(
|
|
"Invalid run state: 'installed_workflow_id' must be a "
|
|
f"string or null, got {type(installed_workflow_id).__name__}"
|
|
)
|
|
if not _ID_PATTERN.fullmatch(installed_workflow_id):
|
|
raise ValueError(
|
|
"Invalid run state: 'installed_workflow_id' must be a "
|
|
"lowercase alphanumeric workflow ID with hyphens"
|
|
)
|
|
if installed_registry_root is not None:
|
|
if not isinstance(installed_registry_root, str):
|
|
raise ValueError(
|
|
"Invalid run state: 'installed_registry_root' must be a "
|
|
f"string or null, got {type(installed_registry_root).__name__}"
|
|
)
|
|
if not installed_registry_root or not Path(
|
|
installed_registry_root
|
|
).is_absolute():
|
|
raise ValueError(
|
|
"Invalid run state: 'installed_registry_root' must be "
|
|
"an absolute path or null"
|
|
)
|
|
if installed_workflow_id is None:
|
|
raise ValueError(
|
|
"Invalid run state: 'installed_registry_root' requires "
|
|
"'installed_workflow_id'"
|
|
)
|
|
|
|
def __init__(
|
|
self,
|
|
run_id: str | None = None,
|
|
workflow_id: str = "",
|
|
project_root: Path | None = None,
|
|
installed_workflow_id: str | None = None,
|
|
installed_registry_root: str | None = None,
|
|
installed_origin_tracked: bool = True,
|
|
) -> None:
|
|
# ``run_id is None`` (omitted) → auto-generate. An explicit empty
|
|
# string is *not* the same as "omitted" and must be validated like
|
|
# any other caller-provided value — otherwise ``__init__("")``
|
|
# would silently substitute a UUID while ``load("")`` rejects, and
|
|
# the two entry points would diverge on the empty-string vector.
|
|
if run_id is None:
|
|
self.run_id = str(uuid.uuid4())[:8]
|
|
else:
|
|
self.run_id = run_id
|
|
self._validate_run_id(self.run_id)
|
|
self._validate_installed_origin(
|
|
installed_workflow_id, installed_registry_root
|
|
)
|
|
self.workflow_id = workflow_id
|
|
self.project_root = project_root or Path(".")
|
|
# Identifies the installed workflow (if any) this run was started
|
|
# from, and the project root that owns its registry — set by
|
|
# execute() when the source was resolved to an installed ID (see
|
|
# workflow_run's ownership mapping). None for a direct/non-installed
|
|
# YAML source. ``installed_origin_tracked`` distinguishes those
|
|
# explicit None values from legacy state files that predate both
|
|
# fields, allowing the CLI to conservatively infer same-project
|
|
# registry ownership before resuming.
|
|
self.installed_workflow_id = installed_workflow_id
|
|
self.installed_registry_root = installed_registry_root
|
|
self.installed_origin_tracked = installed_origin_tracked
|
|
self.status = RunStatus.CREATED
|
|
self.current_step_index = 0
|
|
self.current_step_id: str | None = None
|
|
self.step_results: dict[str, dict[str, Any]] = {}
|
|
# Guards step_results mutation and save() so a concurrent fan-out cannot
|
|
# mutate the dict while save() is serializing it (which would raise
|
|
# "dictionary changed size during iteration").
|
|
self._lock = threading.Lock()
|
|
# Serializes append_log's list append + log.jsonl write so concurrent
|
|
# fan-out workers cannot interleave or corrupt log lines. Kept separate
|
|
# from _lock so frequent logging never contends with state saves; since
|
|
# append_log is never called while _lock is held, the two never nest.
|
|
self._log_lock = threading.Lock()
|
|
self.inputs: dict[str, Any] = {}
|
|
self.created_at = datetime.now(timezone.utc).isoformat()
|
|
self.updated_at = self.created_at
|
|
self.log_entries: list[dict[str, Any]] = []
|
|
|
|
@property
|
|
def runs_dir(self) -> Path:
|
|
return self.project_root / ".specify" / "workflows" / "runs" / self.run_id
|
|
|
|
def record_step_result(self, step_id: str, data: dict[str, Any]) -> None:
|
|
"""Record one step's result under the run lock.
|
|
|
|
Routing the mutation through the lock keeps it from racing a concurrent
|
|
``save()`` that is iterating ``step_results`` (e.g. during a concurrent
|
|
fan-out). For a sequential run this is an uncontended lock.
|
|
"""
|
|
with self._lock:
|
|
self.step_results[step_id] = data
|
|
|
|
def set_step_output(self, step_id: str, output: Any) -> None:
|
|
"""Replace an already-recorded step's ``output`` under the run lock.
|
|
|
|
Fan-out updates its parent step's output after the items have run;
|
|
routing that nested mutation through the lock keeps it from racing a
|
|
``save()`` serializing ``step_results`` — the same invariant
|
|
``record_step_result`` provides for the top-level assignment.
|
|
"""
|
|
with self._lock:
|
|
if step_id in self.step_results:
|
|
self.step_results[step_id]["output"] = output
|
|
|
|
def save(self) -> None:
|
|
"""Persist current state to disk.
|
|
|
|
Held under the run lock and written atomically (temp file + ``os.replace``)
|
|
so a concurrent fan-out can neither mutate ``step_results`` mid-serialization
|
|
nor leave a reader observing a half-written file. Racing writers only
|
|
contend to be last; they never corrupt.
|
|
"""
|
|
runs_dir = self.runs_dir
|
|
runs_dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
with self._lock:
|
|
# Stamp updated_at inside the lock so the timestamp matches the
|
|
# snapshot this thread serializes (concurrent savers don't race it).
|
|
self.updated_at = datetime.now(timezone.utc).isoformat()
|
|
state_data = {
|
|
"run_id": self.run_id,
|
|
"workflow_id": self.workflow_id,
|
|
"installed_workflow_id": self.installed_workflow_id,
|
|
"installed_registry_root": self.installed_registry_root,
|
|
"status": self.status.value,
|
|
"current_step_index": self.current_step_index,
|
|
"current_step_id": self.current_step_id,
|
|
"step_results": self.step_results,
|
|
"created_at": self.created_at,
|
|
"updated_at": self.updated_at,
|
|
}
|
|
self._atomic_write_json(runs_dir / "state.json", state_data)
|
|
self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs})
|
|
|
|
@staticmethod
|
|
def _atomic_write_json(path: Path, data: dict[str, Any]) -> None:
|
|
"""Write *data* as indented JSON to *path* atomically (temp + ``os.replace``)."""
|
|
fd, tmp = tempfile.mkstemp(
|
|
dir=str(path.parent), prefix=f".{path.name}.", suffix=".tmp"
|
|
)
|
|
try:
|
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
|
json.dump(data, f, indent=2)
|
|
os.replace(tmp, path)
|
|
except BaseException:
|
|
try:
|
|
os.unlink(tmp)
|
|
except OSError:
|
|
pass
|
|
raise
|
|
|
|
@classmethod
|
|
def load(cls, run_id: str, project_root: Path) -> RunState:
|
|
"""Load a run state from disk.
|
|
|
|
Validates ``run_id`` against ``_RUN_ID_PATTERN`` *before* building
|
|
the lookup path. Without this guard, a caller passing a value like
|
|
``../escape`` (e.g. via ``specify workflow resume`` CLI argument)
|
|
would interpolate path-traversal segments into
|
|
``runs_dir`` below, letting ``state_path.exists()`` probe arbitrary
|
|
paths and ``json.load`` read attacker-planted JSON from outside
|
|
the project's ``runs/`` directory. ``__init__`` already runs this
|
|
check on the stored ``state_data["run_id"]``, but that fires
|
|
*after* the file lookup — too late to prevent the disclosure.
|
|
Mirrors the precedent in ``agents._ensure_within_directory``.
|
|
"""
|
|
cls._validate_run_id(run_id)
|
|
runs_dir = project_root / ".specify" / "workflows" / "runs" / run_id
|
|
state_path = runs_dir / "state.json"
|
|
if not state_path.exists():
|
|
msg = f"Run state not found: {state_path}"
|
|
raise FileNotFoundError(msg)
|
|
|
|
with open(state_path, encoding="utf-8") as f:
|
|
state_data = json.load(f)
|
|
if not isinstance(state_data, dict):
|
|
raise ValueError("Invalid run state: expected a JSON object")
|
|
missing_fields = [
|
|
field
|
|
for field in ("run_id", "workflow_id", "status")
|
|
if field not in state_data
|
|
]
|
|
if missing_fields:
|
|
raise ValueError(
|
|
"Invalid run state: missing required field(s): "
|
|
+ ", ".join(missing_fields)
|
|
)
|
|
|
|
workflow_id = state_data["workflow_id"]
|
|
if not isinstance(workflow_id, str) or not _ID_PATTERN.fullmatch(
|
|
workflow_id
|
|
):
|
|
raise ValueError(
|
|
"Invalid run state: 'workflow_id' must be a lowercase "
|
|
"alphanumeric workflow ID with hyphens"
|
|
)
|
|
|
|
has_installed_workflow_id = "installed_workflow_id" in state_data
|
|
has_installed_registry_root = "installed_registry_root" in state_data
|
|
if has_installed_workflow_id != has_installed_registry_root:
|
|
raise ValueError(
|
|
"Invalid run state: installed workflow origin fields must "
|
|
"either both be present or both be absent"
|
|
)
|
|
|
|
installed_workflow_id = state_data.get("installed_workflow_id")
|
|
installed_registry_root = state_data.get("installed_registry_root")
|
|
|
|
state = cls(
|
|
run_id=state_data["run_id"],
|
|
workflow_id=workflow_id,
|
|
project_root=project_root,
|
|
installed_workflow_id=installed_workflow_id,
|
|
installed_registry_root=installed_registry_root,
|
|
installed_origin_tracked=has_installed_workflow_id,
|
|
)
|
|
state.status = RunStatus(state_data["status"])
|
|
state.current_step_index = state_data.get("current_step_index", 0)
|
|
state.current_step_id = state_data.get("current_step_id")
|
|
state.step_results = state_data.get("step_results", {})
|
|
state.created_at = state_data.get("created_at", "")
|
|
state.updated_at = state_data.get("updated_at", "")
|
|
|
|
inputs_path = runs_dir / "inputs.json"
|
|
if inputs_path.exists():
|
|
with open(inputs_path, encoding="utf-8") as f:
|
|
inputs_data = json.load(f)
|
|
if not isinstance(inputs_data, dict):
|
|
raise ValueError(
|
|
"Invalid run inputs: expected a JSON object"
|
|
)
|
|
inputs = inputs_data.get("inputs", {})
|
|
if not isinstance(inputs, dict):
|
|
raise ValueError(
|
|
"Invalid run inputs: 'inputs' must be a JSON object"
|
|
)
|
|
state.inputs = inputs
|
|
|
|
return state
|
|
|
|
def append_log(self, entry: dict[str, Any]) -> None:
|
|
"""Append a log entry to the run log.
|
|
|
|
Held under ``_log_lock`` so concurrent fan-out workers serialize their
|
|
list append and ``log.jsonl`` write rather than interleaving lines.
|
|
"""
|
|
entry["timestamp"] = datetime.now(timezone.utc).isoformat()
|
|
runs_dir = self.runs_dir
|
|
runs_dir.mkdir(parents=True, exist_ok=True)
|
|
with self._log_lock:
|
|
self.log_entries.append(entry)
|
|
with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f:
|
|
f.write(json.dumps(entry) + "\n")
|
|
|
|
|
|
# -- Workflow Engine ------------------------------------------------------
|
|
|
|
|
|
class WorkflowEngine:
|
|
"""Orchestrator that loads, validates, and executes workflow definitions."""
|
|
|
|
def __init__(self, project_root: Path | None = None) -> None:
|
|
self.project_root = project_root or Path(".")
|
|
self.on_step_start: Any = None # Callable[[str, str], None] | None
|
|
# Serializes on_step_start so a concurrent fan-out can't interleave the
|
|
# callback's output (the CLI sets it to a console.print lambda). Uncontended
|
|
# for sequential runs.
|
|
self._callback_lock = threading.Lock()
|
|
|
|
def load_workflow(self, source: str | Path) -> WorkflowDefinition:
|
|
"""Load a workflow from an installed ID or a local YAML path.
|
|
|
|
Parameters
|
|
----------
|
|
source:
|
|
Either a workflow ID (looked up in the installed workflows
|
|
directory) or a path to a YAML file.
|
|
|
|
Returns
|
|
-------
|
|
A parsed ``WorkflowDefinition`` (not yet validated; call
|
|
``validate_workflow()`` or ``engine.validate()`` separately).
|
|
|
|
Raises
|
|
------
|
|
FileNotFoundError:
|
|
If the workflow file cannot be found.
|
|
ValueError:
|
|
If the workflow YAML is invalid.
|
|
"""
|
|
path = Path(source).expanduser()
|
|
|
|
# Try as a direct file path first
|
|
if path.suffix.lower() in (".yml", ".yaml") and path.is_file():
|
|
return WorkflowDefinition.from_yaml(path)
|
|
|
|
# Try as an installed workflow ID
|
|
installed_path = (
|
|
self.project_root
|
|
/ ".specify"
|
|
/ "workflows"
|
|
/ str(source)
|
|
/ "workflow.yml"
|
|
)
|
|
if installed_path.exists():
|
|
return WorkflowDefinition.from_yaml(installed_path)
|
|
|
|
msg = f"Workflow not found: {source}"
|
|
raise FileNotFoundError(msg)
|
|
|
|
def validate(self, definition: WorkflowDefinition) -> list[str]:
|
|
"""Validate a workflow definition."""
|
|
return validate_workflow(definition)
|
|
|
|
def execute(
|
|
self,
|
|
definition: WorkflowDefinition,
|
|
inputs: dict[str, Any] | None = None,
|
|
run_id: str | None = None,
|
|
installed_workflow_id: str | None = None,
|
|
installed_registry_root: Path | None = None,
|
|
) -> RunState:
|
|
"""Execute a workflow definition.
|
|
|
|
Parameters
|
|
----------
|
|
definition:
|
|
The validated workflow definition.
|
|
inputs:
|
|
User-provided input values.
|
|
run_id:
|
|
Optional run ID (uses SPECKIT_WORKFLOW_RUN_ID when set, otherwise auto-generated).
|
|
installed_workflow_id, installed_registry_root:
|
|
When the run was started from an installed workflow (as opposed
|
|
to a direct/non-installed YAML source), identifies it and its
|
|
owning registry root so a later ``resume`` can re-check the
|
|
registry's current disabled state before continuing — see
|
|
``workflow_resume``.
|
|
|
|
Returns
|
|
-------
|
|
The final ``RunState`` after execution completes (or pauses).
|
|
"""
|
|
from . import STEP_REGISTRY
|
|
|
|
effective_run_id = run_id
|
|
if effective_run_id is None:
|
|
env_run_id = os.environ.get("SPECKIT_WORKFLOW_RUN_ID", "").strip()
|
|
if env_run_id:
|
|
effective_run_id = env_run_id
|
|
|
|
state = RunState(
|
|
run_id=effective_run_id,
|
|
workflow_id=definition.id,
|
|
project_root=self.project_root,
|
|
installed_workflow_id=installed_workflow_id,
|
|
installed_registry_root=(
|
|
str(installed_registry_root)
|
|
if installed_registry_root is not None
|
|
else None
|
|
),
|
|
)
|
|
|
|
# Persist a copy of the workflow definition so resume can
|
|
# reload it even if the original source is no longer available
|
|
# (e.g. a local YAML path that was moved or deleted).
|
|
run_dir = self.project_root / ".specify" / "workflows" / "runs" / state.run_id
|
|
run_dir.mkdir(parents=True, exist_ok=True)
|
|
workflow_copy = run_dir / "workflow.yml"
|
|
import yaml
|
|
with open(workflow_copy, "w", encoding="utf-8") as f:
|
|
yaml.safe_dump(definition.data, f, sort_keys=False)
|
|
|
|
# Resolve inputs
|
|
resolved_inputs = self._resolve_inputs(definition, inputs or {})
|
|
state.inputs = resolved_inputs
|
|
state.status = RunStatus.RUNNING
|
|
state.save()
|
|
|
|
context = StepContext(
|
|
inputs=resolved_inputs,
|
|
default_integration=definition.default_integration,
|
|
default_model=definition.default_model,
|
|
default_options=definition.default_options,
|
|
project_root=str(self.project_root),
|
|
run_id=state.run_id,
|
|
)
|
|
|
|
# Execute steps
|
|
try:
|
|
self._execute_steps(definition.steps, context, state, STEP_REGISTRY)
|
|
except KeyboardInterrupt:
|
|
state.status = RunStatus.PAUSED
|
|
state.append_log({"event": "workflow_interrupted"})
|
|
state.save()
|
|
return state
|
|
except Exception as exc:
|
|
state.status = RunStatus.FAILED
|
|
state.append_log({"event": "workflow_failed", "error": str(exc)})
|
|
state.save()
|
|
raise
|
|
|
|
if state.status == RunStatus.RUNNING:
|
|
state.status = RunStatus.COMPLETED
|
|
state.append_log({"event": "workflow_finished", "status": state.status.value})
|
|
state.save()
|
|
return state
|
|
|
|
def resume(
|
|
self,
|
|
run_id: str,
|
|
inputs: dict[str, Any] | None = None,
|
|
) -> RunState:
|
|
"""Resume a paused or failed workflow run.
|
|
|
|
When ``inputs`` is provided, the values are merged over the run's
|
|
persisted inputs and re-resolved through the same typed validation
|
|
path used by :meth:`execute`, so the resumed step sees updated
|
|
workflow inputs. Keys not supplied keep their persisted values; an
|
|
empty/``None`` ``inputs`` leaves the run's inputs unchanged.
|
|
"""
|
|
state = RunState.load(run_id, self.project_root)
|
|
if state.status not in (RunStatus.PAUSED, RunStatus.FAILED):
|
|
msg = f"Cannot resume run {run_id!r} with status {state.status.value!r}."
|
|
raise ValueError(msg)
|
|
|
|
# Load the workflow definition — try the persisted copy in the
|
|
# run directory first so resume works even if the original
|
|
# source (e.g. a local YAML path) is no longer available.
|
|
run_dir = self.project_root / ".specify" / "workflows" / "runs" / run_id
|
|
run_copy = run_dir / "workflow.yml"
|
|
if run_copy.exists():
|
|
definition = WorkflowDefinition.from_yaml(run_copy)
|
|
else:
|
|
definition = self.load_workflow(state.workflow_id)
|
|
|
|
# Merge any newly-supplied inputs over the persisted ones and
|
|
# re-validate through the same typing path as the initial run.
|
|
if inputs:
|
|
merged = {**state.inputs, **inputs}
|
|
state.inputs = self._resolve_inputs(definition, merged)
|
|
|
|
# Restore context
|
|
context = StepContext(
|
|
inputs=state.inputs,
|
|
steps=state.step_results,
|
|
default_integration=definition.default_integration,
|
|
default_model=definition.default_model,
|
|
default_options=definition.default_options,
|
|
project_root=str(self.project_root),
|
|
run_id=state.run_id,
|
|
)
|
|
|
|
from . import STEP_REGISTRY
|
|
|
|
state.status = RunStatus.RUNNING
|
|
state.save()
|
|
|
|
# Resume from the current step — re-execute it so gates
|
|
# can prompt interactively again.
|
|
remaining_steps = definition.steps[state.current_step_index :]
|
|
step_offset = state.current_step_index
|
|
|
|
try:
|
|
self._execute_steps(
|
|
remaining_steps, context, state, STEP_REGISTRY,
|
|
step_offset=step_offset,
|
|
)
|
|
except KeyboardInterrupt:
|
|
state.status = RunStatus.PAUSED
|
|
state.append_log({"event": "workflow_interrupted"})
|
|
state.save()
|
|
return state
|
|
except Exception as exc:
|
|
state.status = RunStatus.FAILED
|
|
state.append_log({"event": "resume_failed", "error": str(exc)})
|
|
state.save()
|
|
raise
|
|
|
|
if state.status == RunStatus.RUNNING:
|
|
state.status = RunStatus.COMPLETED
|
|
state.append_log({"event": "workflow_finished", "status": state.status.value})
|
|
state.save()
|
|
return state
|
|
|
|
@staticmethod
|
|
def _record_result(
|
|
context: StepContext, state: RunState, step_id: str, data: dict[str, Any]
|
|
) -> None:
|
|
"""Record a step result into both the live context and persistent state.
|
|
|
|
``record_step_result`` writes ``state.step_results`` under the run lock.
|
|
On a resume run ``context.steps`` *is* that same dict, so that locked
|
|
write is the only one needed; mirror into ``context.steps`` separately
|
|
only when it is a distinct object (a fresh run), to avoid an unlocked
|
|
mutation of the shared dict that could race a concurrent ``save()``.
|
|
"""
|
|
if context.steps is not state.step_results:
|
|
context.steps[step_id] = data
|
|
state.record_step_result(step_id, data)
|
|
|
|
def _execute_steps(
|
|
self,
|
|
steps: list[dict[str, Any]],
|
|
context: StepContext,
|
|
state: RunState,
|
|
registry: dict[str, Any],
|
|
*,
|
|
step_offset: int = 0,
|
|
) -> None:
|
|
"""Execute a list of steps sequentially."""
|
|
for i, step_config in enumerate(steps):
|
|
step_id = step_config.get("id", f"step-{i}")
|
|
step_type = step_config.get("type", "command")
|
|
|
|
state.current_step_id = step_id
|
|
if step_offset >= 0:
|
|
state.current_step_index = step_offset + i
|
|
state.save()
|
|
|
|
state.append_log(
|
|
{"event": "step_started", "step_id": step_id, "type": step_type}
|
|
)
|
|
|
|
# Log progress — use the engine's on_step_start callback if set,
|
|
# otherwise stay silent (library-safe default).
|
|
label = step_config.get("command", "") or step_type
|
|
if self.on_step_start is not None:
|
|
with self._callback_lock:
|
|
self.on_step_start(step_id, label)
|
|
|
|
step_impl = registry.get(step_type)
|
|
if not step_impl:
|
|
state.status = RunStatus.FAILED
|
|
state.append_log(
|
|
{
|
|
"event": "step_failed",
|
|
"step_id": step_id,
|
|
"error": f"Unknown step type: {step_type!r}",
|
|
}
|
|
)
|
|
state.save()
|
|
return
|
|
|
|
result: StepResult = step_impl.execute(step_config, context)
|
|
|
|
# Record step results — prefer resolved values from step output
|
|
step_data = {
|
|
"type": step_type,
|
|
"integration": result.output.get("integration")
|
|
or step_config.get("integration")
|
|
or context.default_integration,
|
|
"model": result.output.get("model")
|
|
or step_config.get("model")
|
|
or context.default_model,
|
|
"options": result.output.get("options")
|
|
or step_config.get("options", {}),
|
|
"input": result.output.get("input")
|
|
or step_config.get("input", {}),
|
|
"output": result.output,
|
|
"status": result.status.value,
|
|
}
|
|
self._record_result(context, state, step_id, step_data)
|
|
|
|
state.append_log(
|
|
{
|
|
"event": "step_completed",
|
|
"step_id": step_id,
|
|
"status": result.status.value,
|
|
}
|
|
)
|
|
|
|
# Handle gate pauses
|
|
if result.status == StepStatus.PAUSED:
|
|
state.status = RunStatus.PAUSED
|
|
state.save()
|
|
return
|
|
|
|
# Handle failures
|
|
if result.status == StepStatus.FAILED:
|
|
# Gate abort (output.aborted) maps to ABORTED status.
|
|
# Aborts are deliberate operator decisions, so
|
|
# `continue_on_error` does NOT override them — that flag
|
|
# is for transient/expected step failures only.
|
|
if result.output.get("aborted"):
|
|
state.status = RunStatus.ABORTED
|
|
state.append_log(
|
|
{
|
|
"event": "workflow_aborted",
|
|
"step_id": step_id,
|
|
}
|
|
)
|
|
state.save()
|
|
return
|
|
|
|
# `continue_on_error: true` lets the pipeline route
|
|
# around the failure instead of halting. The step
|
|
# result (including exit_code, stderr, status) is
|
|
# still recorded so a downstream `if` or `switch`
|
|
# can branch on it (or a `gate` can surface it to the
|
|
# operator via message interpolation). Log a single,
|
|
# unambiguous event per failure resolution — either
|
|
# the run continued past it, or it halted.
|
|
#
|
|
# Use identity comparison (`is True`) rather than
|
|
# truthiness so that only a literal boolean enables
|
|
# the behaviour, even if validation was skipped.
|
|
# Validation rejects non-bool values at parse time,
|
|
# but `WorkflowEngine.execute()` does not auto-validate
|
|
# (see `WorkflowEngine.load_workflow`, whose docstring
|
|
# explicitly notes "not yet validated; call
|
|
# `validate_workflow()` or `engine.validate()`
|
|
# separately"), so a caller passing an unvalidated
|
|
# definition could otherwise see truthy non-bool
|
|
# values like the string `"true"` silently change
|
|
# run semantics.
|
|
if step_config.get("continue_on_error") is True:
|
|
state.append_log(
|
|
{
|
|
"event": "step_continue_on_error",
|
|
"step_id": step_id,
|
|
"error": result.error,
|
|
}
|
|
)
|
|
state.save()
|
|
continue
|
|
|
|
state.status = RunStatus.FAILED
|
|
state.append_log(
|
|
{
|
|
"event": "step_failed",
|
|
"step_id": step_id,
|
|
"error": result.error,
|
|
}
|
|
)
|
|
state.save()
|
|
return
|
|
|
|
# Execute nested steps (from control flow)
|
|
# NOTE: Nested steps run with step_offset=-1 so they don't
|
|
# update current_step_index. If a nested step pauses,
|
|
# resume will re-run the parent step and its nested body.
|
|
# A step-path stack for exact nested resume is a future
|
|
# enhancement.
|
|
if result.next_steps:
|
|
self._execute_steps(
|
|
result.next_steps, context, state, registry,
|
|
step_offset=-1,
|
|
)
|
|
if state.status in (
|
|
RunStatus.PAUSED,
|
|
RunStatus.FAILED,
|
|
RunStatus.ABORTED,
|
|
):
|
|
return
|
|
|
|
# Loop iteration: while/do-while re-evaluate after body
|
|
if step_type in ("while", "do-while"):
|
|
from .expressions import evaluate_condition
|
|
|
|
max_iters = step_config.get("max_iterations")
|
|
# A bool is an int in Python (isinstance(True, int) is True
|
|
# and True == 1), so a bool max_iterations would slip past
|
|
# the int check and cap the loop at range(0)==1 iteration
|
|
# instead of the default. Exclude bools, mirroring the
|
|
# while/do-while validators and the continue_on_error guard.
|
|
if (
|
|
isinstance(max_iters, bool)
|
|
or not isinstance(max_iters, int)
|
|
or max_iters < 1
|
|
):
|
|
max_iters = 10
|
|
condition = step_config.get("condition", False)
|
|
for _loop_iter in range(max_iters - 1):
|
|
if not evaluate_condition(condition, context):
|
|
break
|
|
# Namespace nested step IDs per iteration
|
|
# so logs and state keys are unique.
|
|
# Execute one step at a time and alias each
|
|
# result back to the unprefixed key so that
|
|
# later steps in the same body and the loop
|
|
# condition see the latest values.
|
|
for ns_idx, ns in enumerate(result.next_steps):
|
|
ns_copy = dict(ns)
|
|
orig = ns_copy.get("id")
|
|
base_id = orig or f"step-{ns_idx}"
|
|
ns_copy["id"] = f"{step_id}:{base_id}:{_loop_iter + 1}"
|
|
self._execute_steps(
|
|
[ns_copy], context, state, registry,
|
|
step_offset=-1,
|
|
)
|
|
if state.status in (
|
|
RunStatus.PAUSED,
|
|
RunStatus.FAILED,
|
|
RunStatus.ABORTED,
|
|
):
|
|
return
|
|
if orig and ns_copy["id"] in context.steps:
|
|
self._record_result(
|
|
context, state, orig,
|
|
context.steps[ns_copy["id"]],
|
|
)
|
|
|
|
# Fan-out: execute the nested step template once per item. Honors
|
|
# max_concurrency — <=1 runs sequentially (default, historical
|
|
# behavior); >1 runs up to that many items concurrently. Either way
|
|
# results are assembled in item order under the
|
|
# parentId:templateId:index id grammar.
|
|
if step_type == "fan-out":
|
|
items = result.output.get("items", [])
|
|
template = result.output.get("step_template", {})
|
|
if template and items:
|
|
fan_out_results = self._run_fan_out(
|
|
items, template, step_id, context, state, registry,
|
|
result.output.get("max_concurrency", 1),
|
|
)
|
|
context.item = None
|
|
# Preserve original output and add collected results
|
|
fan_out_output = dict(result.output)
|
|
fan_out_output["results"] = fan_out_results
|
|
# set_step_output updates the recorded dict under the run lock;
|
|
# context.steps[step_id] is that same object, so it reflects the
|
|
# change too — no separate (unlocked) context mutation needed.
|
|
state.set_step_output(step_id, fan_out_output)
|
|
if state.status in (
|
|
RunStatus.PAUSED,
|
|
RunStatus.FAILED,
|
|
RunStatus.ABORTED,
|
|
):
|
|
return
|
|
else:
|
|
# Empty items or no template — normalize output
|
|
result.output["results"] = []
|
|
state.set_step_output(step_id, result.output)
|
|
|
|
def _run_fan_out(
|
|
self,
|
|
items: list[Any],
|
|
template: dict[str, Any],
|
|
step_id: str,
|
|
context: StepContext,
|
|
state: RunState,
|
|
registry: dict[str, Any],
|
|
max_concurrency: Any,
|
|
) -> list[Any]:
|
|
"""Run a fan-out template once per item; return per-item outputs in item order.
|
|
|
|
``max_concurrency`` <= 1 (the default) runs items sequentially, identical
|
|
to the historical fan-out behavior. ``max_concurrency`` > 1 runs items on a
|
|
bounded thread pool using a sliding submission window of that size: at most
|
|
that many items are ever in flight, and no new item is launched once the run
|
|
has reached a halting status, so a halt cannot keep starting queued work.
|
|
|
|
Results are always returned in item order (never completion order). On a
|
|
halt (PAUSED/FAILED/ABORTED) the returned prefix is the items up to and
|
|
including the first item *in item order* whose own execution halted the run
|
|
— identical to the sequential path. Later items that have not yet started
|
|
are cancelled; any already running are allowed to finish but their outputs
|
|
are ignored. Halt is attributed per item from that item's recorded result
|
|
(not the shared run status, which a concurrently-running later item may have
|
|
already flipped), so the prefix never drops the actual halting item.
|
|
|
|
``max_concurrency`` is coerced with ``int()``; a value that cannot be
|
|
coerced (``None``, a non-numeric string, …) or that coerces to <= 1 runs
|
|
sequentially, while a numeric string like ``"4"`` or a float like ``4.0``
|
|
is honored.
|
|
"""
|
|
if not items:
|
|
return []
|
|
|
|
halting = (RunStatus.PAUSED, RunStatus.FAILED, RunStatus.ABORTED)
|
|
try:
|
|
workers = max(1, int(max_concurrency))
|
|
except (TypeError, ValueError):
|
|
workers = 1
|
|
# Never spin up more workers than there is work — bounds a user-controlled
|
|
# max_concurrency from over-allocating threads.
|
|
workers = min(workers, len(items))
|
|
|
|
base_id = template.get("id", "item")
|
|
|
|
def item_id(idx: int) -> str:
|
|
# Per-item ID grammar: parentId:templateId:index.
|
|
return f"{step_id}:{base_id}:{idx}"
|
|
|
|
def run_item(idx: int, item_ctx: StepContext) -> Any:
|
|
item_step = dict(template)
|
|
item_step["id"] = item_id(idx)
|
|
self._execute_steps(
|
|
[item_step], item_ctx, state, registry, step_offset=-1,
|
|
)
|
|
# Read back through the context that was actually executed against,
|
|
# not the outer closure — clearer and robust if StepContext copying
|
|
# ever stops sharing the steps dict by reference.
|
|
return item_ctx.steps.get(item_step["id"], {}).get("output", {})
|
|
|
|
# Sequential path — identical to the historical behavior.
|
|
if workers <= 1:
|
|
results: list[Any] = []
|
|
for item_idx, item_val in enumerate(items):
|
|
context.item = item_val
|
|
results.append(run_item(item_idx, context))
|
|
if state.status in halting:
|
|
break
|
|
return results
|
|
|
|
# Concurrent path — bounded sliding window; results assembled in item order.
|
|
n = len(items)
|
|
slots: list[Any] = [None] * n
|
|
|
|
def run_isolated(idx: int) -> Any:
|
|
# Each item runs against its own context copy so context.item is not
|
|
# clobbered across threads; the shared steps dict is written only on the
|
|
# disjoint parentId:templateId:index key (GIL-safe on distinct keys).
|
|
return run_item(idx, dataclasses.replace(context, item=items[idx]))
|
|
|
|
def item_halt_status(idx: int) -> RunStatus | None:
|
|
# If THIS item's own execution halted the run, return the resulting run
|
|
# status; else None. Decided from the item's own recorded result, not
|
|
# the shared run status, so a later item's concurrent halt is never
|
|
# misattributed here. Mirrors the sequential mapping: PAUSED -> PAUSED;
|
|
# FAILED -> ABORTED when aborted, else FAILED, unless continue_on_error
|
|
# routes around it.
|
|
rec = context.steps.get(item_id(idx))
|
|
if rec is None:
|
|
# Ran but recorded nothing — only when the item failed before
|
|
# record_step_result (e.g. an unknown step type returns early).
|
|
# Every item runs the same template, so the shared run status is
|
|
# this item's own outcome; attribute the halt to it.
|
|
return state.status if state.status in halting else None
|
|
status = rec.get("status")
|
|
if status == StepStatus.PAUSED.value:
|
|
return RunStatus.PAUSED
|
|
if status == StepStatus.FAILED.value:
|
|
out = rec.get("output") or {}
|
|
if out.get("aborted"):
|
|
return RunStatus.ABORTED
|
|
if template.get("continue_on_error") is not True:
|
|
return RunStatus.FAILED
|
|
return None
|
|
|
|
# (halting item index, its run status) once a halt is attributed.
|
|
halt: tuple[int, RunStatus] | None = None
|
|
collected = 0
|
|
with ThreadPoolExecutor(max_workers=workers) as pool:
|
|
futures: dict[int, Future] = {}
|
|
next_submit = 0
|
|
for idx in range(n):
|
|
# Refill the window: keep <= workers in flight, and stop launching
|
|
# new items once the run is halting so a halt cannot keep starting
|
|
# queued work. Already-submitted futures are still collected in
|
|
# item order below.
|
|
while (
|
|
next_submit < n
|
|
and len(futures) < workers
|
|
and state.status not in halting
|
|
):
|
|
futures[next_submit] = pool.submit(run_isolated, next_submit)
|
|
next_submit += 1
|
|
|
|
fut = futures.pop(idx, None)
|
|
if fut is None:
|
|
# Safety net: the window submits indices in order and the loop
|
|
# breaks at the first halting item, so every collected index has
|
|
# an in-flight future. Stop cleanly rather than raise if a future
|
|
# change ever breaks that invariant.
|
|
break
|
|
try:
|
|
slots[idx] = fut.result()
|
|
except Exception:
|
|
# A genuine exception escaping a step (not a normal step
|
|
# FAILED, which sets state.status) must not be masked: cancel
|
|
# outstanding work and re-raise — with a bare ``raise`` so the
|
|
# original traceback is preserved — so the engine marks the run
|
|
# failed instead of reporting a vacuous completion. The pool's
|
|
# __exit__ still joins any already-running workers.
|
|
for other in futures.values():
|
|
other.cancel()
|
|
raise
|
|
collected = idx + 1
|
|
halt_status = item_halt_status(idx)
|
|
if halt_status is not None:
|
|
# First halting item in item order: include it (slots[idx] is
|
|
# already set), record its status, and cancel everything pending.
|
|
halt = (idx, halt_status)
|
|
for other in futures.values():
|
|
other.cancel()
|
|
break
|
|
|
|
if halt is not None:
|
|
halted_at, halted_status = halt
|
|
# A later in-flight item may have overwritten state.status before the
|
|
# pool joined; restore the halting item's own outcome so the final run
|
|
# status matches the sequential semantics.
|
|
state.status = halted_status
|
|
return slots[: halted_at + 1]
|
|
return slots[:collected]
|
|
|
|
def _resolve_inputs(
|
|
self,
|
|
definition: WorkflowDefinition,
|
|
provided: dict[str, Any],
|
|
) -> dict[str, Any]:
|
|
"""Resolve workflow inputs against definitions and provided values."""
|
|
resolved: dict[str, Any] = {}
|
|
for name, input_def in definition.inputs.items():
|
|
if not isinstance(input_def, dict):
|
|
continue
|
|
if name in provided:
|
|
# Resolve sentinels for explicitly-provided values too: a
|
|
# caller passing ``{"integration": "auto"}`` (which the
|
|
# workflow prompt advertises as a valid value) must be
|
|
# treated identically to omitting the input and letting the
|
|
# default flow through, so dispatch never sees the literal
|
|
# sentinel.
|
|
value = self._resolve_default(name, provided[name])
|
|
elif "default" in input_def:
|
|
value = self._resolve_default(name, input_def["default"])
|
|
elif input_def.get("required", False):
|
|
msg = f"Required input {name!r} not provided."
|
|
raise ValueError(msg)
|
|
else:
|
|
continue
|
|
|
|
# When the ``integration`` default could not be resolved against
|
|
# project state and falls back to the literal ``"auto"``
|
|
# sentinel, strip ``enum`` from the input definition before
|
|
# coercion so a workflow that lists specific integrations in
|
|
# ``enum`` does not crash at runtime on the sentinel value.
|
|
# NOTE: only enum-membership is skipped; ``_coerce_input``
|
|
# still enforces the declared ``type`` against the filtered
|
|
# definition (``string`` rejects non-strings, ``number`` rejects
|
|
# bools and uncoercible values, ``boolean`` rejects non-bools),
|
|
# so ill-typed values still fail fast here.
|
|
coerce_input_def = input_def
|
|
if (
|
|
name == "integration"
|
|
and value == "auto"
|
|
and "enum" in input_def
|
|
):
|
|
coerce_input_def = {
|
|
key: val
|
|
for key, val in input_def.items()
|
|
if key != "enum"
|
|
}
|
|
resolved[name] = self._coerce_input(name, value, coerce_input_def)
|
|
return resolved
|
|
|
|
def _resolve_default(self, name: str, default: Any) -> Any:
|
|
"""Resolve special default sentinels against project state.
|
|
|
|
For the ``integration`` input, ``"auto"`` resolves to the integration
|
|
recorded in ``.specify/integration.json`` so workflows dispatch to the
|
|
AI the project was actually initialized with, instead of a hardcoded
|
|
value baked into the workflow YAML.
|
|
"""
|
|
if name == "integration" and default == "auto":
|
|
resolved = self._load_project_integration()
|
|
if resolved is not None:
|
|
return resolved
|
|
return default
|
|
|
|
def _load_project_integration(self) -> str | None:
|
|
"""Read the default integration key from ``.specify/integration.json``.
|
|
|
|
Delegates parsing and schema validation to
|
|
:func:`try_read_integration_json` — the same low-level helper used by
|
|
the CLI — so the engine cannot drift from CLI behavior on the parse
|
|
path. Returns ``None`` when the file is missing, malformed, or
|
|
written by a newer CLI; callers fall back to the literal default.
|
|
"""
|
|
state, error = try_read_integration_json(self.project_root)
|
|
if state is None or error is not None:
|
|
return None
|
|
return default_integration_key(state)
|
|
|
|
@staticmethod
|
|
def _coerce_input(
|
|
name: str, value: Any, input_def: dict[str, Any]
|
|
) -> Any:
|
|
"""Coerce a provided input value to the declared type."""
|
|
input_type = input_def.get("type", "string")
|
|
enum_values = input_def.get("enum")
|
|
|
|
if input_type == "number":
|
|
# Reject bools explicitly: ``bool`` is a subclass of ``int`` so
|
|
# ``float(True)`` succeeds and would silently coerce a YAML
|
|
# authoring mistake like ``type: number`` + ``default: true``
|
|
# into ``1``. Fail fast instead.
|
|
if isinstance(value, bool):
|
|
msg = f"Input {name!r} expected a number, got {value!r}."
|
|
raise ValueError(msg)
|
|
try:
|
|
value = float(value)
|
|
if value == int(value):
|
|
value = int(value)
|
|
except (ValueError, TypeError, OverflowError):
|
|
# OverflowError: `int(value)` raises it for an infinite float
|
|
# (e.g. a `default: .inf` authoring mistake), which would
|
|
# otherwise escape validate_workflow's `except ValueError` and
|
|
# break its "return errors, never raise" contract. Surface it as
|
|
# the same clean "expected a number" error as NaN does.
|
|
msg = f"Input {name!r} expected a number, got {value!r}."
|
|
raise ValueError(msg) from None
|
|
elif input_type == "boolean":
|
|
if isinstance(value, str):
|
|
if value.lower() in ("true", "1", "yes"):
|
|
value = True
|
|
elif value.lower() in ("false", "0", "no"):
|
|
value = False
|
|
else:
|
|
msg = f"Input {name!r} expected a boolean, got {value!r}."
|
|
raise ValueError(msg)
|
|
elif not isinstance(value, bool):
|
|
msg = f"Input {name!r} expected a boolean, got {value!r}."
|
|
raise ValueError(msg)
|
|
elif input_type == "string":
|
|
# Without this, ``type: string`` accepts any Python value
|
|
# (numbers, lists, dicts) because nothing else rejects it —
|
|
# YAML ``default: 5`` would slip through. Require an actual
|
|
# string so authoring mistakes fail at resolve time.
|
|
if not isinstance(value, str):
|
|
msg = f"Input {name!r} expected a string, got {value!r}."
|
|
raise ValueError(msg)
|
|
|
|
if enum_values is not None and value not in enum_values:
|
|
msg = (
|
|
f"Input {name!r} value {value!r} not in allowed "
|
|
f"values: {enum_values}."
|
|
)
|
|
raise ValueError(msg)
|
|
|
|
return value
|
|
|
|
def list_runs(self) -> list[dict[str, Any]]:
|
|
"""List all workflow runs in the project."""
|
|
runs_dir = self.project_root / ".specify" / "workflows" / "runs"
|
|
if not runs_dir.exists():
|
|
return []
|
|
|
|
runs: list[dict[str, Any]] = []
|
|
for run_dir in sorted(runs_dir.iterdir()):
|
|
if not run_dir.is_dir():
|
|
continue
|
|
state_path = run_dir / "state.json"
|
|
if state_path.exists():
|
|
with open(state_path, encoding="utf-8") as f:
|
|
state_data = json.load(f)
|
|
runs.append(state_data)
|
|
return runs
|
|
|
|
|
|
class WorkflowAbortError(Exception):
|
|
"""Raised when a workflow is aborted (e.g., gate rejection)."""
|