mirror of
https://github.com/github/spec-kit.git
synced 2026-07-06 22:32:13 +08:00
fix: resolve GitHub release asset API URL for private repo bundle downloads (#3136)
* fix: resolve GitHub release asset API URL for private repo bundle downloads For private/SSO-protected GitHub repos, browser release download URLs (https://github.com/<owner>/<repo>/releases/download/<tag>/<asset>) redirect to an HTML/SSO page instead of delivering the asset, causing bundle manifest downloads to fail. Extends the pattern from #2855 (presets/workflows) to cover the bundle manifest download path in _download_remote_manifest: - Resolves browser release URLs to GitHub REST API asset URLs via resolve_github_release_asset_api_url before downloading - Direct REST API asset URLs (api.github.com/repos/.../releases/assets/<id>) are passed through directly - Both cases use Accept: application/octet-stream so the API returns the binary payload rather than JSON metadata - The original catalog URL is used to determine artifact format (.zip vs YAML) since the resolved API URL does not carry the file extension Adds two CLI-level contract tests: - bundle info resolves browser release URL via GitHub tags API - bundle info passes direct API asset URL through with octet-stream Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: detect ZIP payload by magic bytes; add zip and API-asset tests Address Copilot review feedback on PR #3136: 1. Detect ZIP payloads by magic bytes (PK\x03\x04) in addition to the '.zip' URL suffix so that direct GitHub REST asset URLs — which carry no file extension — are correctly routed through the ZIP extraction path when the asset is a ZIP bundle artifact. 2. Add two new contract tests: - test_bundle_info_resolves_github_browser_release_url_zip: exercises the '.zip' browser release URL path end-to-end, verifying the tags API lookup fires, octet-stream header is used, and bundle.yml is successfully extracted from the ZIP payload. - test_bundle_info_api_asset_url_zip_detected_by_magic_bytes: verifies that a direct REST asset URL returning ZIP bytes is detected by magic and parsed correctly without a tags API call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: improve error message, broaden ZIP magic, drop unused tmp_path Address second-round Copilot review feedback on PR #3136: - Error message: when the download fails, report the original catalog download_url so the user knows which entry to fix; include the resolved REST API URL when it differs for easier debugging. - ZIP detection: broaden the magic-bytes check from PK\x03\x04 to raw[:2] == b"PK", covering all valid ZIP variants (local-file header PK\x03\x04, empty-archive PK\x05\x06, spanned/split PK\x07\x08). - Tests: remove the unused tmp_path parameter from test_bundle_info_resolves_github_browser_release_url_zip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use full 4-byte ZIP signatures instead of 2-byte PK prefix Address Copilot feedback: raw[:2] == b"PK" is too broad and could misclassify any payload starting with ASCII "PK" as a ZIP, producing a confusing "not a valid bundle" error. Use the three specific 4-byte ZIP magic signatures instead: PK\x03\x04 — local file header (standard ZIP) PK\x05\x06 — end-of-central-directory (empty archive) PK\x07\x08 — data descriptor / spanning marker Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: harden _download_remote_manifest parsing and tighten tests - Promote _ZIP_SIGNATURES to module-level constant (was redefined per call) - Use PurePosixPath for URL path suffix extraction so query strings and fragments are ignored and URL paths are treated as POSIX on all OSes - Move yaml/BundleManifest imports to function top to flatten the previously nested try/except into a single handler with explicit except _yaml.YAMLError and except Exception clauses - Re-add None guard on _local_manifest_source return: the function is typed Optional[BundleManifest] and without the guard a None return propagates silently to callers that degrade gracefully rather than raising an actionable error; comment explains it is defensive not dead - Assert exact resolved asset URL in browser-URL download tests, not just the Accept header, so a regression where download uses the original URL instead of the resolved one would be caught - Add resolution-failure test: when tags API finds no matching asset the code falls back to the original URL and exits non-zero with Error: Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(bundle): pass github_provider_hosts() for GHES private release downloads Extends the GHES support pattern from extensions and presets (#2855, #3157) to the bundle manifest download path: resolve_github_release_asset_api_url now receives github_hosts=github_provider_hosts() so browser release URLs from GitHub Enterprise Server instances are resolved via /api/v3 rather than falling back to the unauthenticated download path. Also adds a contract test covering the GHES resolution path for _download_remote_manifest (analogous to the existing github.com tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(bundle): remove unused ghes_entry variable from GHES contract test The dict was defined but never consumed — the test drives GHES host recognition entirely through the github_provider_hosts() patch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(bundle): include source URL in remote manifest parse errors Thread the catalog URL (and resolved API URL when it differs) into the YAML parse, generic parse, and ZIP-extraction error paths of _download_remote_manifest so failures point at the offending source instead of an opaque temp path. Addresses PR review feedback. Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Manfred Riem <15701806+mnriem@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
lselvar
@@ -631,6 +631,14 @@ def catalog_remove(
|
||||
console.print(f"[green]✓[/green] Removed catalog source '{removed}'.")
|
||||
|
||||
|
||||
# ZIP magic-byte signatures used to detect .zip payloads from REST API asset
|
||||
# URLs, which carry no file extension. The three signatures cover all valid
|
||||
# ZIP variants (PK\x03\x04 = local file header, PK\x05\x06 = empty archive,
|
||||
# PK\x07\x08 = spanning marker) without the false-positive risk of checking
|
||||
# only the 2-byte "PK" prefix.
|
||||
_ZIP_SIGNATURES = (b"PK\x03\x04", b"PK\x05\x06", b"PK\x07\x08")
|
||||
|
||||
|
||||
# ===== internal helpers =====
|
||||
|
||||
|
||||
@@ -794,41 +802,110 @@ def _download_remote_manifest(entry_id: str, url: str):
|
||||
"""Fetch a remote bundle artifact over HTTPS and extract its manifest."""
|
||||
import io
|
||||
import tempfile
|
||||
from pathlib import PurePosixPath
|
||||
from urllib.parse import urlparse as _urlparse
|
||||
|
||||
from ...authentication.http import open_url
|
||||
import yaml as _yaml
|
||||
|
||||
from ...authentication.http import github_provider_hosts, open_url
|
||||
from ..._github_http import resolve_github_release_asset_api_url
|
||||
from ...bundler.models.manifest import BundleManifest
|
||||
|
||||
def _validate_redirect(old_url: str, new_url: str) -> None:
|
||||
_require_https(f"bundle '{entry_id}'", new_url)
|
||||
|
||||
_require_https(f"bundle '{entry_id}'", url)
|
||||
|
||||
# For private/SSO-protected GitHub repos, browser release download URLs
|
||||
# (https://github.com/<owner>/<repo>/releases/download/<tag>/<asset>)
|
||||
# redirect to an HTML/SSO page instead of delivering the asset. Resolve
|
||||
# such URLs to the GitHub REST API asset URL so the authenticated client
|
||||
# can download the actual file.
|
||||
extra_headers = None
|
||||
effective_url = url
|
||||
resolved = resolve_github_release_asset_api_url(
|
||||
url, open_url, timeout=30, github_hosts=github_provider_hosts()
|
||||
)
|
||||
if resolved:
|
||||
effective_url = resolved
|
||||
_require_https(f"bundle '{entry_id}'", effective_url)
|
||||
extra_headers = {"Accept": "application/octet-stream"}
|
||||
|
||||
# Human-readable description of where the bytes came from, reused across
|
||||
# all post-download error messages so failures point at the catalog URL
|
||||
# (and resolved API URL, if any) instead of an opaque temp path.
|
||||
if effective_url != url:
|
||||
_source_desc = f"{url} (resolved to {effective_url})"
|
||||
else:
|
||||
_source_desc = url
|
||||
|
||||
try:
|
||||
with open_url(url, timeout=30, redirect_validator=_validate_redirect) as resp:
|
||||
with open_url(
|
||||
effective_url,
|
||||
timeout=30,
|
||||
redirect_validator=_validate_redirect,
|
||||
extra_headers=extra_headers,
|
||||
) as resp:
|
||||
_require_https(f"bundle '{entry_id}'", resp.geturl())
|
||||
raw = resp.read()
|
||||
except BundlerError:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise BundlerError(f"Failed to download bundle '{entry_id}' from {url}: {exc}") from exc
|
||||
# Report the original catalog URL so users know which entry to fix,
|
||||
# and include the resolved URL when it differs for easier debugging.
|
||||
raise BundlerError(
|
||||
f"Failed to download bundle '{entry_id}' from {_source_desc}: {exc}"
|
||||
) from exc
|
||||
|
||||
# A .zip artifact is written to a temp file and parsed via the local-source
|
||||
# path (which extracts bundle.yml); any other payload is treated as YAML.
|
||||
if url.lower().endswith(".zip"):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
artifact = Path(tmp) / "bundle.zip"
|
||||
artifact.write_bytes(raw)
|
||||
manifest = _local_manifest_source(str(artifact))
|
||||
if manifest is None:
|
||||
raise BundlerError(
|
||||
f"Downloaded artifact for bundle '{entry_id}' is not a valid bundle."
|
||||
)
|
||||
return manifest
|
||||
# Detection uses the path component of the original catalog URL (via
|
||||
# PurePosixPath so query strings and fragments are ignored, and URL paths
|
||||
# are always treated as POSIX regardless of host OS), falling back to the
|
||||
# module-level _ZIP_SIGNATURES magic-byte check for direct REST API asset
|
||||
# URLs which carry no file extension.
|
||||
_url_ext = PurePosixPath(_urlparse(url).path).suffix.lower()
|
||||
try:
|
||||
if _url_ext == ".zip" or raw[:4] in _ZIP_SIGNATURES:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
artifact = Path(tmp) / "bundle.zip"
|
||||
artifact.write_bytes(raw)
|
||||
# Wrap ZIP parsing so any failure (BadZipFile, missing
|
||||
# bundle.yml, etc.) references the source URL rather than the
|
||||
# opaque temporary path, consistent with the download-error
|
||||
# handling above.
|
||||
try:
|
||||
manifest = _local_manifest_source(str(artifact))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise BundlerError(
|
||||
f"Downloaded artifact for bundle '{entry_id}' from "
|
||||
f"{_source_desc} is not a valid bundle: {exc}"
|
||||
) from exc
|
||||
# _local_manifest_source returns None only when the file does
|
||||
# not exist; since we just wrote *artifact* that cannot happen
|
||||
# here. The explicit guard ensures callers never receive None
|
||||
# and silently degrade instead of raising a clear error.
|
||||
if manifest is None:
|
||||
raise BundlerError(
|
||||
f"Downloaded artifact for bundle '{entry_id}' from "
|
||||
f"{_source_desc} is not a valid bundle."
|
||||
)
|
||||
return manifest
|
||||
|
||||
import yaml as _yaml
|
||||
|
||||
from ...bundler.models.manifest import BundleManifest
|
||||
|
||||
data = _yaml.safe_load(io.BytesIO(raw))
|
||||
return BundleManifest.from_dict(data)
|
||||
data = _yaml.safe_load(io.BytesIO(raw))
|
||||
return BundleManifest.from_dict(data)
|
||||
except BundlerError:
|
||||
raise
|
||||
except _yaml.YAMLError as exc:
|
||||
raise BundlerError(
|
||||
f"Downloaded content for bundle '{entry_id}' from {_source_desc} "
|
||||
f"is not valid YAML: {exc}"
|
||||
) from exc
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise BundlerError(
|
||||
f"Failed to parse downloaded bundle '{entry_id}' from "
|
||||
f"{_source_desc}: {exc}"
|
||||
) from exc
|
||||
|
||||
|
||||
def register(app: typer.Typer) -> None:
|
||||
|
||||
Reference in New Issue
Block a user