mirror of
https://github.com/sveltejs/ai-tools.git
synced 2026-07-04 03:19:38 +08:00
Compare commits
5 Commits
init-docs
...
wip-docs-w
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
199d57a8e3 | ||
|
|
47fa0a4382 | ||
|
|
4c6232a44f | ||
|
|
8edbf2f36b | ||
|
|
6e54719f88 |
@@ -1,3 +1,2 @@
|
||||
.claude
|
||||
.github
|
||||
*.test.ts
|
||||
.github
|
||||
108
CLAUDE.md
108
CLAUDE.md
@@ -2,112 +2,72 @@
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Monorepo Structure
|
||||
|
||||
This is a pnpm monorepo containing Svelte MCP (Model Context Protocol) server implementations across multiple packages and applications:
|
||||
|
||||
- **apps/mcp-remote**: SvelteKit web application with MCP server functionality
|
||||
- **packages/mcp-server**: Core MCP server implementation with code analysis tools
|
||||
- **packages/mcp-stdio**: Standalone MCP server CLI with STDIO transport
|
||||
- **packages/mcp-schema**: Shared schema definitions and database utilities
|
||||
|
||||
## Development Commands
|
||||
|
||||
This is a Svelte MCP (Model Context Protocol) server implementation that includes both SvelteKit web interface and MCP server functionality.
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
pnpm i
|
||||
cp apps/mcp-remote/.env.example apps/mcp-remote/.env
|
||||
cp .env.example .env
|
||||
# Set the VOYAGE_API_KEY for embeddings support in .env
|
||||
```
|
||||
|
||||
### Starting the mcp-remote app in dev mode
|
||||
|
||||
```bash
|
||||
# Start the SvelteKit development server for mcp-remote (from root)
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
Or navigate to the app directory:
|
||||
|
||||
```bash
|
||||
cd apps/mcp-remote
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
### Common Commands (from root)
|
||||
|
||||
- `pnpm build` - Build all packages and applications
|
||||
- `pnpm check` - Run type checking across all packages
|
||||
- `pnpm lint` - Run prettier check and eslint across all packages
|
||||
- `pnpm format` - Format code with prettier across all packages
|
||||
- `pnpm test` - Run unit tests across all packages
|
||||
- `pnpm test:watch` - Run tests in watch mode
|
||||
|
||||
### mcp-remote App Commands
|
||||
|
||||
Navigate to `apps/mcp-remote/` to run these commands:
|
||||
### Common Commands
|
||||
|
||||
- `pnpm dev` - Start SvelteKit development server
|
||||
- `pnpm build` - Build the SvelteKit application for production
|
||||
- `pnpm build:mcp` - Build the MCP server TypeScript files
|
||||
- `pnpm build` - Build the application for production
|
||||
- `pnpm start` - Run the MCP server (Node.js entry point)
|
||||
- `pnpm check` - Run Svelte type checking
|
||||
- `pnpm check:watch` - Run type checking in watch mode
|
||||
- `pnpm lint` - Run prettier check and eslint
|
||||
- `pnpm format` - Format code with prettier
|
||||
- `pnpm test` - Run unit tests with vitest
|
||||
- `pnpm test:watch` - Run tests in watch mode
|
||||
|
||||
### Database Commands (Drizzle ORM)
|
||||
|
||||
- `pnpm db:push` - Push schema changes to database
|
||||
- `pnpm db:generate` - Generate migration files
|
||||
- `pnpm db:migrate` - Run migrations
|
||||
- `pnpm db:studio` - Open Drizzle Studio
|
||||
- `pnpm inspect` - Start MCP inspector at http://localhost:6274/
|
||||
|
||||
### MCP Inspector Usage
|
||||
|
||||
After running `pnpm inspect`, visit http://localhost:6274/:
|
||||
- Transport type: `Streamable HTTP`
|
||||
- URL: http://localhost:5173/mcp (when dev server is running)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Monorepo Package Structure
|
||||
### MCP Server Implementation
|
||||
|
||||
- **@sveltejs/mcp-remote**: Full SvelteKit application with web interface and MCP server
|
||||
- **@sveltejs/mcp-server**: Core MCP server logic and code analysis engine (private workspace package)
|
||||
- **@sveltejs/mcp**: Standalone CLI MCP server with STDIO transport (publishable)
|
||||
- **@sveltejs/mcp-schema**: Shared database schema and utilities (private workspace package)
|
||||
The core MCP server is implemented in `src/lib/mcp/index.ts` using the `tmcp` library with:
|
||||
|
||||
### mcp-remote App (apps/mcp-remote)
|
||||
- **Transport Layers**: Both HTTP (`HttpTransport`) and STDIO (`StdioTransport`) support
|
||||
- **Schema Validation**: Uses Valibot with `ValibotJsonSchemaAdapter`
|
||||
- **Main Tool**: `svelte-autofixer` - analyzes Svelte code and provides suggestions/fixes
|
||||
|
||||
The main SvelteKit application that provides both web interface and MCP server functionality:
|
||||
### Code Analysis Engine
|
||||
|
||||
- **Entry Point**: `src/index.js` for Node.js MCP server
|
||||
- **SvelteKit Integration**: `src/hooks.server.ts` integrates MCP HTTP transport with SvelteKit requests
|
||||
- **MCP Server**: `src/lib/mcp/index.ts` - HTTP and STDIO transport support
|
||||
- **Database**: SQLite with Drizzle ORM, vector storage for embeddings
|
||||
- **Content Sync**: `src/lib/server/contentSync.ts` and `src/lib/server/contentDb.ts` for content management
|
||||
Located in `src/lib/server/analyze/`:
|
||||
|
||||
### mcp-server Package (packages/mcp-server)
|
||||
- **Parser** (`parse.ts`): Uses `svelte-eslint-parser` and TypeScript parser to analyze Svelte components
|
||||
- **Scope Analysis**: Tracks variables, references, and scopes across the AST
|
||||
- **Rune Detection**: Identifies Svelte 5 runes (`$state`, `$effect`, `$derived`, etc.)
|
||||
|
||||
Core MCP server implementation shared across applications:
|
||||
### Autofixer System
|
||||
|
||||
- **Main Export**: `src/index.ts`
|
||||
- **MCP Implementation**: `src/mcp/index.ts` using `tmcp` library with Valibot schema validation
|
||||
- **Code Analysis**: Svelte component parsing with `svelte-eslint-parser` and TypeScript parser
|
||||
- **Autofixers**: Visitor pattern implementations for code analysis and suggestions
|
||||
- **Tools**: `svelte-autofixer` - analyzes Svelte code and provides suggestions/fixes
|
||||
- **Autofixers** (`src/lib/mcp/autofixers.ts`): Visitor pattern implementations for code analysis
|
||||
- **Walker Utility** (`src/lib/index.ts`): Enhanced AST walking with visitor mixing capabilities
|
||||
- **Current Autofixer**: `assign_in_effect` - detects assignments to `$state` variables inside `$effect` blocks
|
||||
|
||||
### mcp-stdio Package (packages/mcp-stdio)
|
||||
### Database Layer
|
||||
|
||||
Standalone publishable MCP server with STDIO transport:
|
||||
- **ORM**: Drizzle with SQLite backend
|
||||
- **Schema** (`src/lib/server/db/schema.ts`): Vector table for embeddings support
|
||||
- **Utils** (`src/lib/server/db/utils.ts`): Custom float32 array type for vectors
|
||||
|
||||
- **CLI Binary**: `svelte-mcp` command
|
||||
- **Entry Point**: `src/index.ts`
|
||||
- **Transport**: Uses `@tmcp/transport-stdio` for command-line integration
|
||||
### SvelteKit Integration
|
||||
|
||||
### Database Layer (mcp-remote)
|
||||
|
||||
- **ORM**: Drizzle with SQLite backend (`test.db`)
|
||||
- **Schema**: Located in `src/lib/server/db/schema.ts` with vector table for embeddings
|
||||
- **Configuration**: `drizzle.config.ts` in mcp-remote app
|
||||
- **Hooks** (`src/hooks.server.ts`): Integrates MCP HTTP transport with SvelteKit requests
|
||||
- **Routes**: Basic web interface for the MCP server
|
||||
|
||||
## Key Dependencies
|
||||
|
||||
@@ -121,7 +81,7 @@ Standalone publishable MCP server with STDIO transport:
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
For the mcp-remote app (`apps/mcp-remote/.env`):
|
||||
Required environment variables:
|
||||
|
||||
- `DATABASE_URL`: SQLite database path (default: `file:test.db`)
|
||||
- `VOYAGE_API_KEY`: API key for embeddings support (optional)
|
||||
|
||||
@@ -6,7 +6,7 @@ Repo for the official Svelte MCP server.
|
||||
|
||||
```
|
||||
pnpm i
|
||||
cp apps/mcp-remote/.env.example apps/mcp-remote/.env
|
||||
cp .env.example .env
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
DATABASE_URL=file:test.db
|
||||
DATABASE_TOKEN=needs_to_be_set_but_it_can_be_anything
|
||||
VOYAGE_API_KEY=your_actual_api_key_here
|
||||
VOYAGE_API_KEY=your_actual_api_key_here
|
||||
GITHUB_WEBHOOK_SECRET=some_secret
|
||||
@@ -1,14 +1,12 @@
|
||||
import { defineConfig } from 'drizzle-kit';
|
||||
|
||||
if (!process.env.DATABASE_URL) throw new Error('DATABASE_URL is not set');
|
||||
if (!process.env.DATABASE_TOKEN) throw new Error('DATABASE_TOKEN is not set');
|
||||
|
||||
export default defineConfig({
|
||||
schema: './src/lib/server/db/schema.ts',
|
||||
dialect: 'turso',
|
||||
dbCredentials: {
|
||||
url: process.env.DATABASE_URL,
|
||||
authToken: process.env.DATABASE_TOKEN || '',
|
||||
},
|
||||
dbCredentials: { url: process.env.DATABASE_URL, authToken: process.env.DATABASE_TOKEN },
|
||||
verbose: true,
|
||||
strict: true,
|
||||
});
|
||||
|
||||
@@ -41,7 +41,6 @@
|
||||
"@eslint/js": "^9.36.0",
|
||||
"@libsql/client": "^0.14.0",
|
||||
"@modelcontextprotocol/inspector": "^0.16.7",
|
||||
"@sveltejs/adapter-node": "^5.3.2",
|
||||
"@sveltejs/adapter-vercel": "^5.6.3",
|
||||
"@sveltejs/kit": "^2.22.0",
|
||||
"@sveltejs/vite-plugin-svelte": "^6.0.0",
|
||||
@@ -65,9 +64,8 @@
|
||||
"dependencies": {
|
||||
"@sveltejs/mcp-schema": "workspace:^",
|
||||
"@sveltejs/mcp-server": "workspace:^",
|
||||
"@tmcp/transport-http": "^0.6.2",
|
||||
"@types/tar-stream": "^3.1.4",
|
||||
"minimatch": "^10.0.3",
|
||||
"tar-stream": "^3.1.7"
|
||||
"@tmcp/transport-http": "^0.6.3",
|
||||
"tmcp": "^1.14.0",
|
||||
"valibot": "^1.1.0"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,138 +0,0 @@
|
||||
import { db } from '$lib/server/db';
|
||||
import { cache } from '$lib/server/db/schema';
|
||||
import { and, eq, sql } from 'drizzle-orm';
|
||||
|
||||
export interface CacheEntry {
|
||||
id: number;
|
||||
cache_key: string;
|
||||
data: Buffer;
|
||||
size_bytes: number;
|
||||
expires_at: Date;
|
||||
created_at: Date;
|
||||
}
|
||||
|
||||
export class CacheDbService {
|
||||
private defaultTTL: number;
|
||||
|
||||
constructor(defaultTTLMinutes: number = 60) {
|
||||
this.defaultTTL = defaultTTLMinutes;
|
||||
}
|
||||
|
||||
async get(key: string): Promise<Buffer | null> {
|
||||
try {
|
||||
const result = await db
|
||||
.select({ data: cache.data })
|
||||
.from(cache)
|
||||
.where(and(eq(cache.cache_key, key), sql`${cache.expires_at} > ${new Date()}`))
|
||||
.limit(1);
|
||||
|
||||
if (result.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return result[0].data;
|
||||
} catch (error) {
|
||||
console.error('Error getting cache entry:', error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async set(key: string, data: Buffer, ttlMinutes?: number): Promise<void> {
|
||||
const ttl = ttlMinutes || this.defaultTTL;
|
||||
const expires_at = new Date(Date.now() + ttl * 60 * 1000);
|
||||
const now = new Date();
|
||||
|
||||
try {
|
||||
await db
|
||||
.insert(cache)
|
||||
.values({
|
||||
cache_key: key,
|
||||
data,
|
||||
size_bytes: data.length,
|
||||
expires_at,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
})
|
||||
.onConflictDoUpdate({
|
||||
target: cache.cache_key,
|
||||
set: {
|
||||
data,
|
||||
size_bytes: data.length,
|
||||
expires_at,
|
||||
updated_at: now,
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('Error setting cache entry:', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async delete(key: string): Promise<boolean> {
|
||||
try {
|
||||
const result = await db.delete(cache).where(eq(cache.cache_key, key));
|
||||
return result.rowsAffected > 0;
|
||||
} catch (error) {
|
||||
console.error('Error deleting cache entry:', error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async clear(): Promise<void> {
|
||||
try {
|
||||
await db.delete(cache);
|
||||
} catch (error) {
|
||||
console.error('Error clearing cache:', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async deleteExpired(): Promise<number> {
|
||||
try {
|
||||
const result = await db.delete(cache).where(sql`${cache.expires_at} <= ${new Date()}`);
|
||||
return result.rowsAffected;
|
||||
} catch (error) {
|
||||
console.error('Error deleting expired cache entries:', error);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
async getStatus(): Promise<{ count: number; keys: string[]; totalSizeBytes: number }> {
|
||||
try {
|
||||
const result = await db
|
||||
.select({
|
||||
cache_key: cache.cache_key,
|
||||
size_bytes: cache.size_bytes,
|
||||
})
|
||||
.from(cache)
|
||||
.where(sql`${cache.expires_at} > ${new Date()}`)
|
||||
.orderBy(cache.created_at);
|
||||
|
||||
const keys = result.map((row) => row.cache_key);
|
||||
const totalSizeBytes = result.reduce((sum, row) => sum + row.size_bytes, 0);
|
||||
|
||||
return {
|
||||
count: result.length,
|
||||
keys,
|
||||
totalSizeBytes,
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('Error getting cache status:', error);
|
||||
return { count: 0, keys: [], totalSizeBytes: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
async has(key: string): Promise<boolean> {
|
||||
try {
|
||||
const result = await db
|
||||
.select({ exists: sql`1` })
|
||||
.from(cache)
|
||||
.where(and(eq(cache.cache_key, key), sql`${cache.expires_at} > ${new Date()}`))
|
||||
.limit(1);
|
||||
return result.length > 0;
|
||||
} catch (error) {
|
||||
console.error('Error checking cache entry:', error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,346 +0,0 @@
|
||||
import type { PresetConfig } from '$lib/presets';
|
||||
import { env } from '$env/dynamic/private';
|
||||
import tarStream from 'tar-stream';
|
||||
import { Readable } from 'stream';
|
||||
import { createGunzip } from 'zlib';
|
||||
import { minimatch } from 'minimatch';
|
||||
import { getPresetContent } from './presetCache';
|
||||
import { CacheDbService } from '$lib/cacheDb';
|
||||
import { log, logAlways, logErrorAlways } from '$lib/log';
|
||||
import { cleanTarballPath } from '$lib/utils/pathUtils';
|
||||
|
||||
let cacheService: CacheDbService | null = null;
|
||||
|
||||
function getCacheService(): CacheDbService {
|
||||
if (!cacheService) {
|
||||
cacheService = new CacheDbService();
|
||||
}
|
||||
return cacheService;
|
||||
}
|
||||
|
||||
function sortFilesWithinGroup(files: string[]): string[] {
|
||||
return files.sort((a, b) => {
|
||||
const aPath = a.split('\n')[0].replace('## ', '');
|
||||
const bPath = b.split('\n')[0].replace('## ', '');
|
||||
|
||||
// Check if one path is a parent of the other
|
||||
if (bPath.startsWith(aPath.replace('/index.md', '/'))) return -1;
|
||||
if (aPath.startsWith(bPath.replace('/index.md', '/'))) return 1;
|
||||
|
||||
return aPath.localeCompare(bPath);
|
||||
});
|
||||
}
|
||||
|
||||
export async function fetchRepositoryTarball(owner: string, repo: string): Promise<Buffer> {
|
||||
const cacheKey = `${owner}/${repo}`;
|
||||
const cache = getCacheService();
|
||||
|
||||
const cachedBuffer = await cache.get(cacheKey);
|
||||
if (cachedBuffer) {
|
||||
logAlways(`Using cached tarball for ${cacheKey} from database`);
|
||||
return cachedBuffer;
|
||||
}
|
||||
|
||||
const url = `https://api.github.com/repos/${owner}/${repo}/tarball`;
|
||||
|
||||
logAlways(`Fetching tarball from: ${url}`);
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${env.GITHUB_TOKEN}`,
|
||||
Accept: 'application/vnd.github.v3.raw',
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to fetch tarball: ${response.statusText}`);
|
||||
}
|
||||
|
||||
if (!response.body) {
|
||||
throw new Error('Response body is null');
|
||||
}
|
||||
|
||||
const chunks: Uint8Array[] = [];
|
||||
const reader = response.body.getReader();
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
chunks.push(value);
|
||||
}
|
||||
|
||||
const buffer = Buffer.concat(chunks);
|
||||
|
||||
// Cache the buffer in database with 60 minutes TTL
|
||||
await cache.set(cacheKey, buffer, 60);
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
export async function processMarkdownFromTarball(
|
||||
tarballBuffer: Buffer,
|
||||
presetConfig: PresetConfig,
|
||||
includePathInfo: boolean,
|
||||
): Promise<string[] | { path: string; content: string }[]> {
|
||||
const { glob, ignore = [], minimize = undefined } = presetConfig;
|
||||
|
||||
// Create a Map to store files for each glob pattern while maintaining order
|
||||
const globResults = new Map<string, unknown[]>();
|
||||
const filePathsByPattern = new Map<string, string[]>();
|
||||
glob.forEach((pattern) => {
|
||||
globResults.set(pattern, []);
|
||||
filePathsByPattern.set(pattern, []);
|
||||
});
|
||||
|
||||
const extractStream = tarStream.extract();
|
||||
|
||||
let processedFiles = 0;
|
||||
let matchedFiles = 0;
|
||||
|
||||
extractStream.on('entry', (header, stream, next) => {
|
||||
processedFiles++;
|
||||
let matched = false;
|
||||
|
||||
for (const pattern of glob) {
|
||||
if (shouldIncludeFile(header.name, pattern, ignore)) {
|
||||
matched = true;
|
||||
matchedFiles++;
|
||||
|
||||
if (header.type === 'file') {
|
||||
let content = '';
|
||||
stream.on('data', (chunk) => (content += chunk.toString()));
|
||||
stream.on('end', () => {
|
||||
// Use the unified path utility to clean tarball paths
|
||||
const cleanPath = cleanTarballPath(header.name);
|
||||
|
||||
const processedContent = minimizeContent(content, minimize);
|
||||
|
||||
if (includePathInfo) {
|
||||
const files = globResults.get(pattern) || [];
|
||||
files.push({
|
||||
path: cleanPath,
|
||||
content: processedContent,
|
||||
});
|
||||
globResults.set(pattern, files);
|
||||
} else {
|
||||
const contentWithHeader = `## ${cleanPath}\n\n${processedContent}`;
|
||||
|
||||
const files = globResults.get(pattern) || [];
|
||||
files.push(contentWithHeader);
|
||||
globResults.set(pattern, files);
|
||||
}
|
||||
|
||||
const paths = filePathsByPattern.get(pattern) || [];
|
||||
paths.push(cleanPath);
|
||||
filePathsByPattern.set(pattern, paths);
|
||||
|
||||
next();
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!matched) {
|
||||
stream.resume();
|
||||
next();
|
||||
}
|
||||
});
|
||||
|
||||
const tarballStream = Readable.from(tarballBuffer);
|
||||
const gunzipStream = createGunzip();
|
||||
|
||||
tarballStream.pipe(gunzipStream).pipe(extractStream);
|
||||
|
||||
await new Promise<void>((resolve) => extractStream.on('finish', resolve));
|
||||
|
||||
logAlways(`Total files processed: ${processedFiles}`);
|
||||
logAlways(`Files matching glob: ${matchedFiles}`);
|
||||
log('\nFinal file order:');
|
||||
|
||||
glob.forEach((pattern, index) => {
|
||||
const paths = filePathsByPattern.get(pattern) || [];
|
||||
const sortedPaths = includePathInfo
|
||||
? paths
|
||||
: sortFilesWithinGroup(paths.map((p) => `## ${p}`)).map((p) => p.replace('## ', ''));
|
||||
|
||||
if (sortedPaths.length > 0) {
|
||||
log(`\nGlob pattern ${index + 1}: ${pattern}`);
|
||||
sortedPaths.forEach((path, i) => {
|
||||
log(` ${i + 1}. ${path}`);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Combine results in the order of glob patterns
|
||||
const orderedResults: unknown[] = [];
|
||||
for (const pattern of glob) {
|
||||
const filesForPattern = globResults.get(pattern) || [];
|
||||
if (includePathInfo) {
|
||||
orderedResults.push(...filesForPattern);
|
||||
} else {
|
||||
orderedResults.push(...sortFilesWithinGroup(filesForPattern as string[]));
|
||||
}
|
||||
}
|
||||
|
||||
return orderedResults as string[] | { path: string; content: string }[];
|
||||
}
|
||||
|
||||
function shouldIncludeFile(filename: string, glob: string, ignore: string[] = []): boolean {
|
||||
const shouldIgnore = ignore.some((pattern) => minimatch(filename, pattern));
|
||||
if (shouldIgnore) {
|
||||
logAlways(`❌ Ignored by pattern: ${filename}`);
|
||||
return false;
|
||||
}
|
||||
|
||||
return minimatch(filename, glob);
|
||||
}
|
||||
|
||||
export async function clearRepositoryCache(): Promise<void> {
|
||||
const cache = getCacheService();
|
||||
await cache.clear();
|
||||
logAlways('Repository cache cleared');
|
||||
}
|
||||
|
||||
export async function getRepositoryCacheStatus(): Promise<{
|
||||
size: number;
|
||||
repositories: string[];
|
||||
totalSizeBytes: number;
|
||||
}> {
|
||||
const cache = getCacheService();
|
||||
const status = await cache.getStatus();
|
||||
return {
|
||||
size: status.count,
|
||||
repositories: status.keys,
|
||||
totalSizeBytes: status.totalSizeBytes,
|
||||
};
|
||||
}
|
||||
|
||||
export interface MinimizeOptions {
|
||||
normalizeWhitespace?: boolean;
|
||||
removeLegacy?: boolean;
|
||||
removePlaygroundLinks?: boolean;
|
||||
removePrettierIgnore?: boolean;
|
||||
removeNoteBlocks?: boolean;
|
||||
removeDetailsBlocks?: boolean;
|
||||
removeHtmlComments?: boolean;
|
||||
removeDiffMarkers?: boolean;
|
||||
}
|
||||
|
||||
const defaultOptions: MinimizeOptions = {
|
||||
normalizeWhitespace: false,
|
||||
removeLegacy: false,
|
||||
removePlaygroundLinks: false,
|
||||
removePrettierIgnore: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: false,
|
||||
removeDiffMarkers: true,
|
||||
};
|
||||
|
||||
function removeQuoteBlocks(content: string, blockType: string): string {
|
||||
return content
|
||||
.split('\n')
|
||||
.reduce((acc: string[], line: string, index: number, lines: string[]) => {
|
||||
// If we find a block (with or without additional text), skip it and all subsequent blockquote lines
|
||||
if (line.trim().startsWith(`> [!${blockType}]`)) {
|
||||
// Skip all subsequent lines that are part of the blockquote
|
||||
let i = index;
|
||||
while (i < lines.length && (lines[i].startsWith('>') || lines[i].trim() === '')) {
|
||||
i++;
|
||||
}
|
||||
// Update the index to skip all these lines
|
||||
index = i - 1;
|
||||
return acc;
|
||||
}
|
||||
|
||||
acc.push(line);
|
||||
return acc;
|
||||
}, [])
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
function removeDiffMarkersFromContent(content: string): string {
|
||||
let inCodeBlock = false;
|
||||
const lines = content.split('\n');
|
||||
const processedLines = lines.map((line) => {
|
||||
// Track if we're entering or leaving a code block
|
||||
// eslint-disable-next-line no-useless-escape
|
||||
if (line.trim().startsWith('\`\`\`')) {
|
||||
inCodeBlock = !inCodeBlock;
|
||||
return line;
|
||||
}
|
||||
|
||||
if (inCodeBlock) {
|
||||
// Handle lines that end with --- or +++ with possible whitespace after
|
||||
// eslint-disable-next-line no-useless-escape
|
||||
line = line.replace(/(\+{3}|\-{3})[\s]*$/g, '');
|
||||
|
||||
// Handle triple markers at start while preserving indentation
|
||||
// This captures the whitespace before the marker and adds it back
|
||||
// eslint-disable-next-line no-useless-escape
|
||||
line = line.replace(/^(\s*)(\+{3}|\-{3})\s*/g, '$1');
|
||||
|
||||
// Handle single + or - markers at start while preserving indentation
|
||||
// eslint-disable-next-line no-useless-escape
|
||||
line = line.replace(/^(\s*)[\+\-](\s)/g, '$1');
|
||||
|
||||
// Handle multi-line diff blocks where --- or +++ might be in the middle of line
|
||||
// eslint-disable-next-line no-useless-escape
|
||||
line = line.replace(/[\s]*(\+{3}|\-{3})[\s]*/g, '');
|
||||
}
|
||||
|
||||
return line;
|
||||
});
|
||||
|
||||
return processedLines.join('\n');
|
||||
}
|
||||
|
||||
export function minimizeContent(content: string, options?: Partial<MinimizeOptions>): string {
|
||||
const settings: MinimizeOptions = options ? { ...defaultOptions, ...options } : defaultOptions;
|
||||
|
||||
let minimized = content;
|
||||
|
||||
minimized = minimized.replace(/NOTE: do not edit this file, it is generated in.*$/gm, '');
|
||||
|
||||
if (settings.removeDiffMarkers) {
|
||||
minimized = removeDiffMarkersFromContent(minimized);
|
||||
}
|
||||
|
||||
if (settings.removeLegacy) {
|
||||
minimized = removeQuoteBlocks(minimized, 'LEGACY');
|
||||
}
|
||||
|
||||
if (settings.removeNoteBlocks) {
|
||||
minimized = removeQuoteBlocks(minimized, 'NOTE');
|
||||
}
|
||||
|
||||
if (settings.removeDetailsBlocks) {
|
||||
minimized = removeQuoteBlocks(minimized, 'DETAILS');
|
||||
}
|
||||
|
||||
if (settings.removePlaygroundLinks) {
|
||||
// Replace playground URLs with /[link] but keep the original link text
|
||||
minimized = minimized.replace(/\[([^\]]+)\]\(\/playground[^)]+\)/g, '[$1](/REMOVED)');
|
||||
}
|
||||
|
||||
if (settings.removePrettierIgnore) {
|
||||
minimized = minimized
|
||||
.split('\n')
|
||||
.filter((line) => line.trim() !== '<!-- prettier-ignore -->')
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
if (settings.removeHtmlComments) {
|
||||
// Replace all HTML comments (including multi-line) with empty string
|
||||
minimized = minimized.replace(/<!--[\s\S]*?-->/g, '');
|
||||
}
|
||||
|
||||
if (settings.normalizeWhitespace) {
|
||||
minimized = minimized.replace(/\s+/g, ' ');
|
||||
}
|
||||
|
||||
minimized = minimized.trim();
|
||||
|
||||
return minimized;
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
import { dev } from '$app/environment';
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const log = (...props: unknown[]) => {
|
||||
if (dev) {
|
||||
console.log(...props);
|
||||
}
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const logWarning = (...props: unknown[]) => {
|
||||
if (dev) {
|
||||
console.warn(...props);
|
||||
}
|
||||
};
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const logError = (...props: unknown[]) => {
|
||||
if (dev) {
|
||||
console.error(...props);
|
||||
}
|
||||
};
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const logAlways = (...props: unknown[]) => {
|
||||
console.log(...props);
|
||||
};
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const logWarningAlways = (...props: unknown[]) => {
|
||||
console.warn(...props);
|
||||
};
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const logErrorAlways = (...props: unknown[]) => {
|
||||
console.error(...props);
|
||||
};
|
||||
@@ -1,213 +0,0 @@
|
||||
import { ContentSyncService } from '$lib/server/contentSync';
|
||||
import { presets } from '$lib/presets';
|
||||
import { log, logAlways, logErrorAlways } from '$lib/log';
|
||||
import { cleanDocumentationPath } from '$lib/utils/pathUtils';
|
||||
import { CacheDbService } from '$lib/cacheDb';
|
||||
|
||||
// Maximum age of cached content in milliseconds (24 hours)
|
||||
export const MAX_CACHE_AGE_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
let cacheService: CacheDbService | null = null;
|
||||
|
||||
function getCacheService(): CacheDbService {
|
||||
if (!cacheService) {
|
||||
cacheService = new CacheDbService();
|
||||
}
|
||||
return cacheService;
|
||||
}
|
||||
|
||||
export async function getPresetContent(presetKey: string): Promise<string | null> {
|
||||
try {
|
||||
const preset = presets[presetKey];
|
||||
if (!preset) {
|
||||
log(`Preset not found: ${presetKey}`);
|
||||
return null;
|
||||
}
|
||||
|
||||
// Check cache first
|
||||
const cache = getCacheService();
|
||||
const cacheKey = `preset:${presetKey}`;
|
||||
|
||||
try {
|
||||
const cachedData = await cache.get(cacheKey);
|
||||
if (cachedData) {
|
||||
const cachedContent = cachedData.toString('utf8');
|
||||
logAlways(`Using cached content for preset ${presetKey}`);
|
||||
return cachedContent;
|
||||
}
|
||||
} catch (cacheError) {
|
||||
logErrorAlways(`Error reading cache for preset ${presetKey}:`, cacheError);
|
||||
// Continue with normal flow if cache read fails
|
||||
}
|
||||
|
||||
// Try to get files from the content table first
|
||||
let filesWithPaths = await ContentSyncService.getPresetContentFromDb(presetKey);
|
||||
|
||||
// If no content in database, fetch from GitHub and sync
|
||||
if (!filesWithPaths || filesWithPaths.length === 0) {
|
||||
logAlways(`No content in database for preset ${presetKey}, fetching from GitHub...`);
|
||||
|
||||
// Sync the repository first
|
||||
await ContentSyncService.syncRepository();
|
||||
|
||||
// Try again from database
|
||||
filesWithPaths = await ContentSyncService.getPresetContentFromDb(presetKey);
|
||||
|
||||
if (!filesWithPaths || filesWithPaths.length === 0) {
|
||||
log(`Still no content found for preset: ${presetKey} after sync`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Format files with headers and preserve the order from database
|
||||
// The files are already correctly ordered by glob pattern precedence
|
||||
// Use the unified path utility to clean paths
|
||||
const files = filesWithPaths.map((f) => {
|
||||
const cleanPath = cleanDocumentationPath(f.path);
|
||||
return `## ${cleanPath}\n\n${f.content}`;
|
||||
});
|
||||
|
||||
// DO NOT sort - files are already in correct glob pattern order from ContentSyncService
|
||||
const content = files.join('\n\n');
|
||||
|
||||
logAlways(`Generated content for ${presetKey} on-demand (${filesWithPaths.length} files)`);
|
||||
|
||||
// Cache the generated content for 1 hour (60 minutes)
|
||||
try {
|
||||
const contentBuffer = Buffer.from(content, 'utf8');
|
||||
await cache.set(cacheKey, contentBuffer, 60); // 60 minutes TTL
|
||||
logAlways(`Cached content for preset ${presetKey} (expires in 1 hour)`);
|
||||
} catch (cacheError) {
|
||||
logErrorAlways(`Error caching content for preset ${presetKey}:`, cacheError);
|
||||
// Don't fail the request if caching fails
|
||||
}
|
||||
|
||||
return content;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error generating preset content for ${presetKey}:`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function getPresetSizeKb(presetKey: string): Promise<number | null> {
|
||||
try {
|
||||
const content = await getPresetContent(presetKey);
|
||||
if (!content) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const sizeKb = Math.floor(new TextEncoder().encode(content).length / 1024);
|
||||
return sizeKb;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error calculating preset size for ${presetKey}:`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function isPresetStale(presetKey: string): Promise<boolean> {
|
||||
try {
|
||||
// Check if the repository content is stale
|
||||
return await ContentSyncService.isRepositoryContentStale();
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error checking preset staleness for ${presetKey}:`, error);
|
||||
return true; // On error, assume stale
|
||||
}
|
||||
}
|
||||
|
||||
export async function presetExists(presetKey: string): Promise<boolean> {
|
||||
try {
|
||||
const preset = presets[presetKey];
|
||||
if (!preset) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// A preset "exists" if it's defined in presets.ts
|
||||
// The content will be generated on-demand
|
||||
return true;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error checking preset existence for ${presetKey}:`, error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function getPresetMetadata(presetKey: string): Promise<{
|
||||
size_kb: number;
|
||||
document_count: number;
|
||||
updated_at: Date;
|
||||
is_stale: boolean;
|
||||
} | null> {
|
||||
try {
|
||||
const preset = presets[presetKey];
|
||||
if (!preset) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Try to get files from content table or GitHub
|
||||
const content = await getPresetContent(presetKey);
|
||||
if (!content) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Get the files again to count them (this will use cached data)
|
||||
const filesWithPaths = await ContentSyncService.getPresetContentFromDb(presetKey);
|
||||
const documentCount = filesWithPaths?.length || 0;
|
||||
|
||||
const sizeKb = Math.floor(new TextEncoder().encode(content).length / 1024);
|
||||
const isStale = await isPresetStale(presetKey);
|
||||
|
||||
return {
|
||||
size_kb: sizeKb,
|
||||
document_count: documentCount,
|
||||
updated_at: new Date(), // Since it's generated on-demand, it's always "now"
|
||||
is_stale: isStale,
|
||||
};
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error getting preset metadata for ${presetKey}:`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear the cache for a specific preset
|
||||
*/
|
||||
export async function clearPresetCache(presetKey: string): Promise<boolean> {
|
||||
try {
|
||||
const cache = getCacheService();
|
||||
const cacheKey = `preset:${presetKey}`;
|
||||
const success = await cache.delete(cacheKey);
|
||||
|
||||
if (success) {
|
||||
logAlways(`Cleared cache for preset ${presetKey}`);
|
||||
}
|
||||
|
||||
return success;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error clearing cache for preset ${presetKey}:`, error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear cache for all presets
|
||||
*/
|
||||
export async function clearAllPresetCaches(): Promise<number> {
|
||||
try {
|
||||
const cache = getCacheService();
|
||||
const allPresetKeys = Object.keys(presets);
|
||||
let clearedCount = 0;
|
||||
|
||||
for (const presetKey of allPresetKeys) {
|
||||
const cacheKey = `preset:${presetKey}`;
|
||||
const success = await cache.delete(cacheKey);
|
||||
if (success) {
|
||||
clearedCount++;
|
||||
}
|
||||
}
|
||||
|
||||
logAlways(`Cleared cache for ${clearedCount} presets`);
|
||||
return clearedCount;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error clearing all preset caches:`, error);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
@@ -1,267 +0,0 @@
|
||||
import type { MinimizeOptions } from './fetchMarkdown';
|
||||
import { SVELTE_5_PROMPT } from '$lib/utils/prompts';
|
||||
|
||||
export type PresetConfig = {
|
||||
title: string;
|
||||
description?: string;
|
||||
glob: string[];
|
||||
ignore?: string[];
|
||||
prompt?: string;
|
||||
minimize?: MinimizeOptions;
|
||||
distilled?: boolean;
|
||||
distilledFilenameBase?: string;
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
export const combinedPresets: Record<string, PresetConfig> = {
|
||||
'svelte-complete-distilled': {
|
||||
title: '🔮 Svelte + SvelteKit (Recommended - LLM Distilled)',
|
||||
description: 'AI-condensed version of the docs focused on code examples and key concepts',
|
||||
glob: [
|
||||
// Svelte
|
||||
'**/apps/svelte.dev/content/docs/svelte/**/*.md',
|
||||
// SvelteKit
|
||||
'**/apps/svelte.dev/content/docs/kit/**/*.md',
|
||||
],
|
||||
minimize: {
|
||||
normalizeWhitespace: false,
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removePrettierIgnore: true,
|
||||
removeNoteBlocks: false,
|
||||
removeDetailsBlocks: false,
|
||||
removeHtmlComments: true,
|
||||
removeDiffMarkers: true,
|
||||
},
|
||||
ignore: [
|
||||
// Svelte ignores (same as medium preset)
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/04-custom-elements.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/06-v4-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/07-v5-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/99-faq.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/xx-reactivity-indepth.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/21-svelte-legacy.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/99-legacy/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/**/*.md',
|
||||
'**/xx-*.md',
|
||||
// SvelteKit ignores (same as medium preset)
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/*adapter-*.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/99-writing-adapters.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/30-advanced/70-packaging.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/05-performance.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/10-accessibility.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/60-appendix/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/98-reference/**/*.md',
|
||||
'**/xx-*.md',
|
||||
],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
distilled: true,
|
||||
distilledFilenameBase: 'svelte-complete-distilled',
|
||||
},
|
||||
'svelte-complete-medium': {
|
||||
title: '⭐️ Svelte + SvelteKit (Medium preset)',
|
||||
description:
|
||||
'Complete Svelte + SvelteKit docs excluding certain advanced sections, legacy, notes and migration docs',
|
||||
glob: [
|
||||
// Svelte
|
||||
'**/apps/svelte.dev/content/docs/svelte/**/*.md',
|
||||
// SvelteKit
|
||||
'**/apps/svelte.dev/content/docs/kit/**/*.md',
|
||||
],
|
||||
ignore: [
|
||||
// Svelte ignores
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/04-custom-elements.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/06-v4-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/07-v5-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/99-faq.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/xx-reactivity-indepth.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/21-svelte-legacy.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/99-legacy/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-runtime-errors.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-runtime-warnings.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-compiler-errors.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-compiler-warnings.md',
|
||||
'**/xx-*.md',
|
||||
// SvelteKit ignores
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/*adapter-*.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/99-writing-adapters.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/30-advanced/70-packaging.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/05-performance.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/10-accessibility.md', // May the a11y gods have mercy on our souls
|
||||
'**/apps/svelte.dev/content/docs/kit/60-appendix/**/*.md',
|
||||
'**/xx-*.md',
|
||||
],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
},
|
||||
'svelte-complete': {
|
||||
title: 'Svelte + SvelteKit (Large preset)',
|
||||
description: 'Complete Svelte + SvelteKit docs excluding legacy, notes and migration docs',
|
||||
glob: [
|
||||
'**/apps/svelte.dev/content/docs/svelte/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/**/*.md',
|
||||
],
|
||||
ignore: [],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
},
|
||||
'svelte-complete-tiny': {
|
||||
title: 'Svelte + SvelteKit (Tiny preset)',
|
||||
description: 'Tutorial content only',
|
||||
glob: [
|
||||
'**/apps/svelte.dev/content/tutorial/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/02-runes/**/*.md',
|
||||
],
|
||||
ignore: [],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
},
|
||||
'svelte-migration': {
|
||||
title: 'Svelte + SvelteKit migration guide',
|
||||
description: 'Only Svelte + SvelteKit docs for migrating ',
|
||||
glob: [
|
||||
// Svelte
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/07-v5-migration-guide.md',
|
||||
// SvelteKit
|
||||
'**/apps/svelte.dev/content/docs/kit/60-appendix/30-migrating-to-sveltekit-2.md',
|
||||
],
|
||||
ignore: [],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
export const sveltePresets: Record<string, PresetConfig> = {
|
||||
svelte: {
|
||||
title: 'Svelte (Full)',
|
||||
description: 'Complete documentation including legacy and reference',
|
||||
glob: ['**/apps/svelte.dev/content/docs/svelte/**/*.md'],
|
||||
ignore: [],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {},
|
||||
},
|
||||
'svelte-medium': {
|
||||
title: 'Svelte (Medium)',
|
||||
description: 'Complete documentation including legacy and reference',
|
||||
glob: ['**/apps/svelte.dev/content/docs/svelte/**/*.md'],
|
||||
ignore: [
|
||||
// Svelte ignores
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/04-custom-elements.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/06-v4-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/07-v5-migration-guide.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/99-faq.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/07-misc/xx-reactivity-indepth.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/21-svelte-legacy.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/99-legacy/**/*.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-runtime-errors.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-runtime-warnings.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-compiler-errors.md',
|
||||
'**/apps/svelte.dev/content/docs/svelte/98-reference/30-compiler-warnings.md',
|
||||
],
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
export const svelteKitPresets: Record<string, PresetConfig> = {
|
||||
sveltekit: {
|
||||
title: 'SvelteKit (Full)',
|
||||
description: 'Complete documentation including legacy and reference',
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
glob: ['**/apps/svelte.dev/content/docs/kit/**/*.md'],
|
||||
minimize: {},
|
||||
},
|
||||
'sveltekit-medium': {
|
||||
title: 'SvelteKit (Medium)',
|
||||
description: 'Complete documentation including legacy and reference',
|
||||
prompt: SVELTE_5_PROMPT,
|
||||
glob: ['**/apps/svelte.dev/content/docs/kit/**/*.md'],
|
||||
minimize: {
|
||||
removeLegacy: true,
|
||||
removePlaygroundLinks: true,
|
||||
removeNoteBlocks: true,
|
||||
removeDetailsBlocks: true,
|
||||
removeHtmlComments: true,
|
||||
normalizeWhitespace: true,
|
||||
},
|
||||
ignore: [
|
||||
// SvelteKit ignores
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/*adapter-*.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/25-build-and-deploy/99-writing-adapters.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/30-advanced/70-packaging.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/05-performance.md',
|
||||
'**/apps/svelte.dev/content/docs/kit/40-best-practices/10-accessibility.md', // May the a11y gods have mercy on our souls
|
||||
'**/apps/svelte.dev/content/docs/kit/60-appendix/**/*.md',
|
||||
'**/xx-*.md',
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
export const otherPresets: Record<string, PresetConfig> = {
|
||||
'svelte-cli': {
|
||||
title: 'Svelte CLI - npx sv',
|
||||
glob: ['**/apps/svelte.dev/content/docs/cli/**/*.md'],
|
||||
ignore: [],
|
||||
minimize: {},
|
||||
},
|
||||
};
|
||||
|
||||
export const presets = {
|
||||
...combinedPresets,
|
||||
...sveltePresets,
|
||||
...svelteKitPresets,
|
||||
...otherPresets,
|
||||
};
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
export function transformAndSortPresets(presetsObject: Record<string, PresetConfig>) {
|
||||
return Object.entries(presetsObject)
|
||||
.map(([key, value]) => ({
|
||||
key: key.toLowerCase(),
|
||||
...value,
|
||||
}))
|
||||
.sort();
|
||||
}
|
||||
|
||||
export const DEFAULT_REPOSITORY = {
|
||||
owner: 'sveltejs',
|
||||
repo: 'svelte.dev',
|
||||
} as const;
|
||||
32
apps/mcp-remote/src/lib/schemas/index.ts
Normal file
32
apps/mcp-remote/src/lib/schemas/index.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
import * as v from 'valibot';
|
||||
|
||||
// not the full schema but it contains the information we need
|
||||
export const github_webhook_schema = v.object({
|
||||
action: v.union([v.literal('closed')]),
|
||||
pull_request: v.object({
|
||||
patch_url: v.string(),
|
||||
merged: v.boolean(),
|
||||
user: v.object({
|
||||
login: v.string(),
|
||||
}),
|
||||
}),
|
||||
});
|
||||
|
||||
export const github_content_schema = v.object({
|
||||
name: v.string(),
|
||||
path: v.string(),
|
||||
sha: v.string(),
|
||||
size: v.number(),
|
||||
url: v.string(),
|
||||
html_url: v.string(),
|
||||
git_url: v.string(),
|
||||
download_url: v.nullable(v.string()),
|
||||
type: v.literal('file'),
|
||||
content: v.string(),
|
||||
encoding: v.literal('base64'),
|
||||
_links: v.object({
|
||||
self: v.string(),
|
||||
git: v.string(),
|
||||
html: v.string(),
|
||||
}),
|
||||
});
|
||||
@@ -1,383 +0,0 @@
|
||||
import { query } from '$lib/server/db';
|
||||
import type {
|
||||
DbContent,
|
||||
DbContentDistilled,
|
||||
CreateContentInput,
|
||||
ContentFilter,
|
||||
ContentStats,
|
||||
} from '$lib/types/db';
|
||||
import { logAlways, logErrorAlways } from '$lib/log';
|
||||
|
||||
// Type mapping for table names to their corresponding types
|
||||
type TableTypeMap = {
|
||||
content: DbContent;
|
||||
content_distilled: DbContentDistilled;
|
||||
};
|
||||
|
||||
// Union type for valid table names
|
||||
type TableName = keyof TableTypeMap;
|
||||
|
||||
export class ContentDbService {
|
||||
static extractFilename(path: string): string {
|
||||
return path.split('/').pop() || path;
|
||||
}
|
||||
|
||||
static async upsertContent(input: CreateContentInput): Promise<DbContent> {
|
||||
try {
|
||||
const result = await query(
|
||||
`INSERT INTO content (
|
||||
path, filename, content, size_bytes, metadata
|
||||
) VALUES ($1, $2, $3, $4, $5)
|
||||
ON CONFLICT (path) DO UPDATE SET
|
||||
content = EXCLUDED.content,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
metadata = EXCLUDED.metadata,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
RETURNING *`,
|
||||
[
|
||||
input.path,
|
||||
input.filename,
|
||||
input.content,
|
||||
input.size_bytes,
|
||||
input.metadata ? JSON.stringify(input.metadata) : '{}',
|
||||
],
|
||||
);
|
||||
|
||||
logAlways(`Upserted content for ${input.path}`);
|
||||
return result.rows[0] as DbContent;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to upsert content for ${input.path}:`, error);
|
||||
throw new Error(
|
||||
`Failed to upsert content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async getContentByPath(path: string): Promise<DbContent | null> {
|
||||
try {
|
||||
const result = await query('SELECT * FROM content WHERE path = $1', [path]);
|
||||
return result.rows.length > 0 ? (result.rows[0] as DbContent) : null;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to get content ${path}:`, error);
|
||||
throw new Error(
|
||||
`Failed to get content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async getAllContent(): Promise<DbContent[]> {
|
||||
try {
|
||||
const result = await query('SELECT * FROM content ORDER BY path');
|
||||
return result.rows as DbContent[];
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to get all content:', error);
|
||||
throw new Error(
|
||||
`Failed to get content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generic search method that works with both content and content_distilled tables
|
||||
*/
|
||||
static async searchContent<T extends TableName>(
|
||||
searchQuery: string,
|
||||
tableName: T,
|
||||
pathPattern: string = 'apps/svelte.dev/content/docs/%',
|
||||
): Promise<TableTypeMap[T] | null> {
|
||||
try {
|
||||
const lowerQuery = searchQuery.toLowerCase();
|
||||
|
||||
// Build table-specific WHERE clauses
|
||||
let baseWhereClause = '';
|
||||
let params: (string | number)[] = [];
|
||||
let paramIndex = 1;
|
||||
|
||||
if (tableName === 'content') {
|
||||
// For content table, include path filter
|
||||
baseWhereClause = `WHERE path LIKE $${paramIndex}`;
|
||||
params = [pathPattern];
|
||||
paramIndex = 2;
|
||||
} else {
|
||||
// For content_distilled table, no additional filters needed
|
||||
baseWhereClause = '';
|
||||
paramIndex = 1;
|
||||
}
|
||||
|
||||
// First, try exact title match using JSON operators
|
||||
const exactTitleQueryStr = `
|
||||
SELECT * FROM ${tableName}
|
||||
${baseWhereClause}${baseWhereClause ? ' AND' : 'WHERE'} LOWER(metadata->>'title') = $${paramIndex}
|
||||
LIMIT 1
|
||||
`;
|
||||
|
||||
const exactTitleParams = [...params, lowerQuery];
|
||||
const exactTitleResult = await query(exactTitleQueryStr, exactTitleParams);
|
||||
|
||||
if (exactTitleResult.rows.length > 0) {
|
||||
return exactTitleResult.rows[0] as TableTypeMap[T];
|
||||
}
|
||||
|
||||
// Then try partial title match
|
||||
const partialTitleQueryStr = `
|
||||
SELECT * FROM ${tableName}
|
||||
${baseWhereClause}${baseWhereClause ? ' AND' : 'WHERE'} LOWER(metadata->>'title') LIKE $${paramIndex}
|
||||
LIMIT 1
|
||||
`;
|
||||
|
||||
const partialTitleParams = [...params, `%${lowerQuery}%`];
|
||||
const partialTitleResult = await query(partialTitleQueryStr, partialTitleParams);
|
||||
|
||||
if (partialTitleResult.rows.length > 0) {
|
||||
return partialTitleResult.rows[0] as TableTypeMap[T];
|
||||
}
|
||||
|
||||
// Finally try path match for backward compatibility
|
||||
const pathMatchQueryStr = `
|
||||
SELECT * FROM ${tableName}
|
||||
${baseWhereClause}${baseWhereClause ? ' AND' : 'WHERE'} LOWER(path) LIKE $${paramIndex}
|
||||
LIMIT 1
|
||||
`;
|
||||
|
||||
const pathMatchParams = [...params, `%${lowerQuery}%`];
|
||||
const pathMatchResult = await query(pathMatchQueryStr, pathMatchParams);
|
||||
|
||||
return pathMatchResult.rows.length > 0 ? (pathMatchResult.rows[0] as TableTypeMap[T]) : null;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to search ${tableName} for "${searchQuery}":`, error);
|
||||
throw new Error(
|
||||
`Failed to search ${tableName}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async searchAllContent(
|
||||
searchQuery: string,
|
||||
pathPattern: string = 'apps/svelte.dev/content/docs/%',
|
||||
limit: number = 50,
|
||||
): Promise<DbContent[]> {
|
||||
try {
|
||||
const lowerQuery = searchQuery.toLowerCase();
|
||||
|
||||
// Combine all search types into one query with UNION
|
||||
const combinedQueryStr = `
|
||||
-- Exact title matches first
|
||||
(SELECT * FROM content
|
||||
WHERE path LIKE $1
|
||||
AND LOWER(metadata->>'title') = $2
|
||||
ORDER BY path
|
||||
LIMIT $3)
|
||||
|
||||
UNION
|
||||
|
||||
-- Then partial title matches
|
||||
(SELECT * FROM content
|
||||
WHERE path LIKE $1
|
||||
AND LOWER(metadata->>'title') LIKE $4
|
||||
AND LOWER(metadata->>'title') != $2
|
||||
ORDER BY path
|
||||
LIMIT $3)
|
||||
|
||||
UNION
|
||||
|
||||
-- Finally path matches
|
||||
(SELECT * FROM content
|
||||
WHERE path LIKE $1
|
||||
AND LOWER(path) LIKE $4
|
||||
AND (metadata->>'title' IS NULL OR LOWER(metadata->>'title') NOT LIKE $4)
|
||||
ORDER BY path
|
||||
LIMIT $3)
|
||||
|
||||
ORDER BY path
|
||||
LIMIT $3
|
||||
`;
|
||||
|
||||
const params = [pathPattern, lowerQuery, limit, `%${lowerQuery}%`];
|
||||
|
||||
const result = await query(combinedQueryStr, params);
|
||||
|
||||
return result.rows as DbContent[];
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to search all content for "${searchQuery}":`, error);
|
||||
throw new Error(
|
||||
`Failed to search content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async getDocumentationSections(
|
||||
pathPattern: string = 'apps/svelte.dev/content/docs/%',
|
||||
minContentLength: number = 100,
|
||||
): Promise<Array<{ path: string; metadata: Record<string, unknown>; content: string }>> {
|
||||
try {
|
||||
const sectionsQueryStr = `
|
||||
SELECT path, metadata, content
|
||||
FROM content
|
||||
WHERE path LIKE $1
|
||||
AND LENGTH(content) >= $2
|
||||
ORDER BY path
|
||||
`;
|
||||
|
||||
const params = [pathPattern, minContentLength];
|
||||
|
||||
const result = await query(sectionsQueryStr, params);
|
||||
|
||||
return result.rows.map((row) => ({
|
||||
path: row.path,
|
||||
metadata: row.metadata,
|
||||
content: row.content,
|
||||
}));
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to get documentation sections:', error);
|
||||
throw new Error(
|
||||
`Failed to get sections: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async getFilteredContent(
|
||||
pathPattern: string = 'apps/svelte.dev/content/docs/%',
|
||||
minContentLength: number = 200,
|
||||
): Promise<DbContent[]> {
|
||||
try {
|
||||
const filterQueryStr = `
|
||||
SELECT *
|
||||
FROM content
|
||||
WHERE path LIKE $1
|
||||
AND LENGTH(content) >= $2
|
||||
ORDER BY path
|
||||
`;
|
||||
|
||||
const params = [pathPattern, minContentLength];
|
||||
|
||||
const result = await query(filterQueryStr, params);
|
||||
return result.rows as DbContent[];
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to get filtered content:', error);
|
||||
throw new Error(
|
||||
`Failed to get filtered content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async getContentStats(): Promise<ContentStats> {
|
||||
try {
|
||||
const totalResult = await query(
|
||||
`SELECT
|
||||
COUNT(*) as total_files,
|
||||
COALESCE(SUM(size_bytes), 0) as total_size_bytes,
|
||||
MAX(updated_at) as last_updated
|
||||
FROM content`,
|
||||
);
|
||||
|
||||
return {
|
||||
total_files: parseInt(totalResult.rows[0].total_files),
|
||||
total_size_bytes: parseInt(totalResult.rows[0].total_size_bytes),
|
||||
last_updated: totalResult.rows[0].last_updated,
|
||||
};
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to get content stats:', error);
|
||||
throw new Error(
|
||||
`Failed to get stats: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async deleteContent(path: string): Promise<boolean> {
|
||||
try {
|
||||
const result = await query('DELETE FROM content WHERE path = $1', [path]);
|
||||
return (result.rowCount ?? 0) > 0;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to delete content ${path}:`, error);
|
||||
throw new Error(
|
||||
`Failed to delete content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async deleteAllContent(): Promise<number> {
|
||||
try {
|
||||
const result = await query('DELETE FROM content');
|
||||
return result.rowCount ?? 0;
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to delete all content:', error);
|
||||
throw new Error(
|
||||
`Failed to delete content: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static async hasContentChanged(path: string, newContent: string): Promise<boolean> {
|
||||
try {
|
||||
const existing = await ContentDbService.getContentByPath(path);
|
||||
if (!existing) return true;
|
||||
|
||||
return existing.content !== newContent;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to check content change for ${path}:`, error);
|
||||
return true; // Assume changed on error
|
||||
}
|
||||
}
|
||||
|
||||
static async batchUpsertContent(contents: CreateContentInput[]): Promise<DbContent[]> {
|
||||
try {
|
||||
const results: DbContent[] = [];
|
||||
|
||||
// Process in chunks to avoid overwhelming the database
|
||||
const chunkSize = 200;
|
||||
for (let i = 0; i < contents.length; i += chunkSize) {
|
||||
const chunk = contents.slice(i, i + chunkSize);
|
||||
|
||||
const chunkResults = await Promise.all(
|
||||
chunk.map((content) => ContentDbService.upsertContent(content)),
|
||||
);
|
||||
|
||||
results.push(...chunkResults);
|
||||
}
|
||||
|
||||
logAlways(`Batch upserted ${results.length} content items`);
|
||||
return results;
|
||||
} catch (error) {
|
||||
logErrorAlways('Failed to batch upsert content:', error);
|
||||
throw new Error(
|
||||
`Failed to batch upsert: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static extractFrontmatter(content: string): Record<string, unknown> {
|
||||
const metadata: Record<string, unknown> = {};
|
||||
|
||||
if (!content.startsWith('---\n')) {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
const endIndex = content.indexOf('\n---\n', 4);
|
||||
if (endIndex === -1) {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
const frontmatter = content.substring(4, endIndex);
|
||||
const lines = frontmatter.split('\n');
|
||||
|
||||
for (const line of lines) {
|
||||
const colonIndex = line.indexOf(':');
|
||||
if (colonIndex > 0) {
|
||||
const key = line.substring(0, colonIndex).trim();
|
||||
const value = line.substring(colonIndex + 1).trim();
|
||||
|
||||
// Remove quotes if present
|
||||
const cleanValue = value.replace(/^["'](.*)["']$/, '$1');
|
||||
|
||||
// Try to parse as JSON for nested structures
|
||||
try {
|
||||
metadata[key] = JSON.parse(cleanValue);
|
||||
} catch {
|
||||
metadata[key] = cleanValue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return metadata;
|
||||
}
|
||||
}
|
||||
@@ -1,271 +0,0 @@
|
||||
import {
|
||||
fetchRepositoryTarball,
|
||||
processMarkdownFromTarball,
|
||||
minimizeContent,
|
||||
} from '$lib/fetchMarkdown';
|
||||
import { ContentDbService } from '$lib/server/contentDb';
|
||||
import type { CreateContentInput } from '$lib/types/db';
|
||||
import { presets, DEFAULT_REPOSITORY } from '$lib/presets';
|
||||
import { logAlways, logErrorAlways, log } from '$lib/log';
|
||||
|
||||
function sortFilesWithinGroup(
|
||||
files: Array<{ path: string; content: string }>,
|
||||
): Array<{ path: string; content: string }> {
|
||||
return files.sort((a, b) => {
|
||||
const aPath = a.path;
|
||||
const bPath = b.path;
|
||||
|
||||
// Check if one path is a parent of the other
|
||||
if (bPath.startsWith(aPath.replace('/index.md', '/'))) return -1;
|
||||
if (aPath.startsWith(bPath.replace('/index.md', '/'))) return 1;
|
||||
|
||||
return aPath.localeCompare(bPath);
|
||||
});
|
||||
}
|
||||
|
||||
export class ContentSyncService {
|
||||
static readonly MAX_CONTENT_AGE_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
static async syncRepository(
|
||||
options: {
|
||||
returnStats?: boolean;
|
||||
} = {},
|
||||
): Promise<{
|
||||
success: boolean;
|
||||
stats: {
|
||||
total_files: number;
|
||||
total_size_bytes: number;
|
||||
last_updated: Date;
|
||||
};
|
||||
sync_details: {
|
||||
upserted_files: number;
|
||||
deleted_files: number;
|
||||
unchanged_files: number;
|
||||
};
|
||||
timestamp: string;
|
||||
}> {
|
||||
const { returnStats = true } = options;
|
||||
const { owner, repo: repoName } = DEFAULT_REPOSITORY;
|
||||
|
||||
logAlways(`Starting sync for repository: ${owner}/${repoName}`);
|
||||
|
||||
let upsertedFiles = 0;
|
||||
let deletedFiles = 0;
|
||||
let unchangedFiles = 0;
|
||||
|
||||
try {
|
||||
logAlways(`Step 1: Syncing repository ${owner}/${repoName}`);
|
||||
|
||||
const tarballBuffer = await fetchRepositoryTarball(owner, repoName);
|
||||
|
||||
const filesWithPaths = (await processMarkdownFromTarball(
|
||||
tarballBuffer,
|
||||
{
|
||||
glob: ['**/*.md', '**/*.mdx'],
|
||||
ignore: [],
|
||||
title: `Sync ${owner}/${repoName}`,
|
||||
distilled: false,
|
||||
},
|
||||
true,
|
||||
)) as Array<{
|
||||
path: string;
|
||||
content: string;
|
||||
}>;
|
||||
|
||||
logAlways(`Found ${filesWithPaths.length} markdown files in ${owner}/${repoName}`);
|
||||
|
||||
const existingFiles = await ContentDbService.getAllContent();
|
||||
const existingPaths = new Set(existingFiles.map((file) => file.path));
|
||||
|
||||
const foundPaths = new Set(filesWithPaths.map((file) => file.path));
|
||||
|
||||
const contentInputs: CreateContentInput[] = [];
|
||||
|
||||
for (const file of filesWithPaths) {
|
||||
const filename = ContentDbService.extractFilename(file.path);
|
||||
const sizeBytes = new TextEncoder().encode(file.content).length;
|
||||
|
||||
const metadata = ContentDbService.extractFrontmatter(file.content);
|
||||
|
||||
const hasChanged = await ContentDbService.hasContentChanged(file.path, file.content);
|
||||
|
||||
if (hasChanged) {
|
||||
contentInputs.push({
|
||||
path: file.path,
|
||||
filename,
|
||||
content: file.content,
|
||||
size_bytes: sizeBytes,
|
||||
metadata,
|
||||
});
|
||||
} else {
|
||||
unchangedFiles++;
|
||||
}
|
||||
}
|
||||
|
||||
if (contentInputs.length > 0) {
|
||||
logAlways(`Upserting ${contentInputs.length} changed files`);
|
||||
await ContentDbService.batchUpsertContent(contentInputs);
|
||||
upsertedFiles = contentInputs.length;
|
||||
} else {
|
||||
logAlways(`No file content changes detected`);
|
||||
}
|
||||
|
||||
// Handle deletions - find files in DB that are no longer in the repository
|
||||
const deletedPaths = Array.from(existingPaths).filter((path) => !foundPaths.has(path));
|
||||
|
||||
if (deletedPaths.length > 0) {
|
||||
logAlways(`Deleting ${deletedPaths.length} files that no longer exist`);
|
||||
|
||||
for (const deletedPath of deletedPaths) {
|
||||
logAlways(` Deleting: ${deletedPath}`);
|
||||
await ContentDbService.deleteContent(deletedPath);
|
||||
}
|
||||
deletedFiles = deletedPaths.length;
|
||||
} else {
|
||||
logAlways(`No deleted files detected`);
|
||||
}
|
||||
|
||||
let stats;
|
||||
if (returnStats) {
|
||||
logAlways(`Step 2: Collecting final statistics`);
|
||||
stats = await ContentSyncService.getContentStats();
|
||||
} else {
|
||||
logAlways(`Step 2: Skipping stats collection (returnStats = false)`);
|
||||
// Return minimal stats structure
|
||||
stats = {
|
||||
total_files: 0,
|
||||
total_size_bytes: 0,
|
||||
last_updated: new Date(),
|
||||
};
|
||||
}
|
||||
|
||||
logAlways(
|
||||
`Sync completed successfully: ${upsertedFiles} upserted, ${deletedFiles} deleted, ${unchangedFiles} unchanged`,
|
||||
);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
stats,
|
||||
sync_details: {
|
||||
upserted_files: upsertedFiles,
|
||||
deleted_files: deletedFiles,
|
||||
unchanged_files: unchangedFiles,
|
||||
},
|
||||
timestamp: new Date().toISOString(),
|
||||
};
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to sync repository ${owner}/${repoName}:`, error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
static async isRepositoryContentStale(): Promise<boolean> {
|
||||
try {
|
||||
const stats = await ContentDbService.getContentStats();
|
||||
|
||||
if (stats.total_files === 0) {
|
||||
return true; // No content, consider stale
|
||||
}
|
||||
|
||||
const lastUpdated = new Date(stats.last_updated);
|
||||
const contentAge = Date.now() - lastUpdated.getTime();
|
||||
|
||||
const isStale = contentAge > ContentSyncService.MAX_CONTENT_AGE_MS;
|
||||
|
||||
if (isStale) {
|
||||
logAlways(
|
||||
`Repository content is stale (age: ${Math.floor(contentAge / 1000 / 60)} minutes)`,
|
||||
);
|
||||
}
|
||||
|
||||
return isStale;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Error checking repository staleness:`, error);
|
||||
return true; // On error, assume stale
|
||||
}
|
||||
}
|
||||
|
||||
static async getPresetContentFromDb(
|
||||
presetKey: string,
|
||||
): Promise<Array<{ path: string; content: string }> | null> {
|
||||
const preset = presets[presetKey];
|
||||
if (!preset) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
const allContent = await ContentDbService.getAllContent();
|
||||
|
||||
if (allContent.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
log(`Checking ${allContent.length} files against glob patterns for preset ${presetKey}`);
|
||||
log(`Glob patterns: ${JSON.stringify(preset.glob)}`);
|
||||
log(`Ignore patterns: ${JSON.stringify(preset.ignore || [])}`);
|
||||
|
||||
const { minimatch } = await import('minimatch');
|
||||
|
||||
const orderedResults: Array<{ path: string; content: string }> = [];
|
||||
|
||||
// Process one glob pattern at a time
|
||||
for (const pattern of preset.glob) {
|
||||
log(`\nProcessing glob pattern: ${pattern}`);
|
||||
|
||||
const matchingFiles: Array<{ path: string; content: string }> = [];
|
||||
|
||||
for (const dbContent of allContent) {
|
||||
const shouldIgnore = preset.ignore?.some((ignorePattern) => {
|
||||
const matches = minimatch(dbContent.path, ignorePattern);
|
||||
if (matches) {
|
||||
log(` File ${dbContent.path} ignored by pattern: ${ignorePattern}`);
|
||||
}
|
||||
return matches;
|
||||
});
|
||||
if (shouldIgnore) continue;
|
||||
|
||||
if (minimatch(dbContent.path, pattern)) {
|
||||
log(` File ${dbContent.path} matched`);
|
||||
|
||||
let processedContent = dbContent.content;
|
||||
if (preset.minimize && Object.keys(preset.minimize).length > 0) {
|
||||
processedContent = minimizeContent(dbContent.content, preset.minimize);
|
||||
}
|
||||
|
||||
matchingFiles.push({
|
||||
path: dbContent.path,
|
||||
content: processedContent,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const sortedFiles = sortFilesWithinGroup(matchingFiles);
|
||||
|
||||
log(` Found ${sortedFiles.length} files for pattern: ${pattern}`);
|
||||
sortedFiles.forEach((file, i) => {
|
||||
log(` ${i + 1}. ${file.path}`);
|
||||
});
|
||||
|
||||
orderedResults.push(...sortedFiles);
|
||||
}
|
||||
|
||||
logAlways(
|
||||
`Found ${orderedResults.length} files matching preset ${presetKey} from database in natural glob order`,
|
||||
);
|
||||
|
||||
log('\nFinal file order:');
|
||||
orderedResults.forEach((file, i) => {
|
||||
log(` ${i + 1}. ${file.path}`);
|
||||
});
|
||||
|
||||
return orderedResults;
|
||||
} catch (error) {
|
||||
logErrorAlways(`Failed to get preset content from database for ${presetKey}:`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
static async getContentStats() {
|
||||
return ContentDbService.getContentStats();
|
||||
}
|
||||
}
|
||||
@@ -1,112 +0,0 @@
|
||||
export interface QueryConfig {
|
||||
debug?: boolean;
|
||||
}
|
||||
|
||||
// Enum for distillable preset names
|
||||
export enum DistillablePreset {
|
||||
SVELTE_DISTILLED = 'svelte-distilled',
|
||||
SVELTEKIT_DISTILLED = 'sveltekit-distilled',
|
||||
SVELTE_COMPLETE_DISTILLED = 'svelte-complete-distilled',
|
||||
}
|
||||
|
||||
// Database table types
|
||||
|
||||
export interface DbDistillation {
|
||||
id: number;
|
||||
preset_name: DistillablePreset;
|
||||
version: string; // 'latest' or '2024-01-15'
|
||||
content: string;
|
||||
size_kb: number;
|
||||
document_count: number;
|
||||
distillation_job_id: number | null;
|
||||
created_at: Date;
|
||||
}
|
||||
|
||||
export interface DbDistillationJob {
|
||||
id: number;
|
||||
preset_name: string;
|
||||
batch_id: string | null;
|
||||
status: 'pending' | 'processing' | 'completed' | 'failed';
|
||||
model_used: string;
|
||||
total_files: number;
|
||||
processed_files: number;
|
||||
successful_files: number;
|
||||
minimize_applied: boolean;
|
||||
total_input_tokens: number;
|
||||
total_output_tokens: number;
|
||||
started_at: Date | null;
|
||||
completed_at: Date | null;
|
||||
error_message: string | null;
|
||||
metadata: Record<string, unknown>; // JSONB
|
||||
created_at: Date;
|
||||
updated_at: Date;
|
||||
}
|
||||
|
||||
export interface DbContent {
|
||||
id: number;
|
||||
path: string;
|
||||
filename: string;
|
||||
content: string;
|
||||
size_bytes: number;
|
||||
metadata: Record<string, unknown>;
|
||||
created_at: Date;
|
||||
updated_at: Date;
|
||||
}
|
||||
|
||||
export interface DbContentDistilled {
|
||||
id: number;
|
||||
path: string;
|
||||
filename: string;
|
||||
content: string;
|
||||
size_bytes: number;
|
||||
metadata: Record<string, unknown>;
|
||||
created_at: Date;
|
||||
updated_at: Date;
|
||||
}
|
||||
|
||||
// Input types for creating/updating records
|
||||
|
||||
export interface CreateDistillationInput {
|
||||
preset_name: DistillablePreset;
|
||||
version: string;
|
||||
content: string;
|
||||
size_kb: number;
|
||||
document_count: number;
|
||||
distillation_job_id?: number;
|
||||
}
|
||||
|
||||
export interface CreateDistillationJobInput {
|
||||
preset_name: string;
|
||||
batch_id?: string;
|
||||
status: 'pending' | 'processing' | 'completed' | 'failed';
|
||||
model_used: string;
|
||||
total_files: number;
|
||||
minimize_applied?: boolean;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CreateContentInput {
|
||||
path: string;
|
||||
filename: string;
|
||||
content: string;
|
||||
size_bytes: number;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CreateContentDistilledInput {
|
||||
path: string;
|
||||
filename: string;
|
||||
content: string;
|
||||
size_bytes: number;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface ContentFilter {
|
||||
path_pattern?: string; // For glob pattern matching
|
||||
}
|
||||
|
||||
export interface ContentStats {
|
||||
total_files: number;
|
||||
total_size_bytes: number;
|
||||
last_updated: Date;
|
||||
}
|
||||
@@ -1,335 +0,0 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
cleanDocumentationPath,
|
||||
cleanTarballPath,
|
||||
extractTitleFromPath,
|
||||
removeFrontmatter,
|
||||
} from './pathUtils.js';
|
||||
|
||||
describe('pathUtils', () => {
|
||||
describe('cleanDocumentationPath', () => {
|
||||
it('should remove apps/svelte.dev/content/ prefix', () => {
|
||||
const input = 'apps/svelte.dev/content/docs/svelte/01-introduction.md';
|
||||
const expected = 'docs/svelte/01-introduction.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle paths without the prefix', () => {
|
||||
const input = 'docs/svelte/01-introduction.md';
|
||||
const expected = 'docs/svelte/01-introduction.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle empty string', () => {
|
||||
const input = '';
|
||||
const expected = '';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle partial prefix matches', () => {
|
||||
const input = 'apps/svelte.dev/content-extra/docs/svelte/01-introduction.md';
|
||||
const expected = 'apps/svelte.dev/content-extra/docs/svelte/01-introduction.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle paths with similar but different prefixes', () => {
|
||||
const input = 'apps/svelte.dev/contents/docs/svelte/01-introduction.md';
|
||||
const expected = 'apps/svelte.dev/contents/docs/svelte/01-introduction.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle SvelteKit documentation paths', () => {
|
||||
const input = 'apps/svelte.dev/content/docs/kit/01-routing.md';
|
||||
const expected = 'docs/kit/01-routing.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle tutorial paths', () => {
|
||||
const input = 'apps/svelte.dev/content/tutorial/01-introduction/01-hello-world.md';
|
||||
const expected = 'tutorial/01-introduction/01-hello-world.md';
|
||||
expect(cleanDocumentationPath(input)).toBe(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('cleanTarballPath', () => {
|
||||
it('should remove the first segment from tarball paths', () => {
|
||||
const input = 'svelte.dev-main/apps/svelte.dev/content/docs/svelte/01-introduction.md';
|
||||
const expected = 'apps/svelte.dev/content/docs/svelte/01-introduction.md';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle paths with different repo prefixes', () => {
|
||||
const input = 'svelte-12345/apps/svelte.dev/content/docs/kit/01-routing.md';
|
||||
const expected = 'apps/svelte.dev/content/docs/kit/01-routing.md';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle single segment paths', () => {
|
||||
const input = 'single-segment';
|
||||
const expected = '';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle empty string', () => {
|
||||
const input = '';
|
||||
const expected = '';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle paths with no segments', () => {
|
||||
const input = 'just-filename.md';
|
||||
const expected = '';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle complex nested paths', () => {
|
||||
const input = 'repo-name/very/deep/nested/path/to/file.md';
|
||||
const expected = 'very/deep/nested/path/to/file.md';
|
||||
expect(cleanTarballPath(input)).toBe(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('extractTitleFromPath', () => {
|
||||
it('should extract filename and remove .md extension and numbered prefix', () => {
|
||||
const input = 'docs/svelte/01-introduction.md';
|
||||
const expected = 'introduction';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should remove numbered prefixes', () => {
|
||||
const input = 'docs/svelte/01-introduction.md';
|
||||
const expected = 'introduction';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle files without numbered prefixes', () => {
|
||||
const input = 'docs/svelte/reactivity.md';
|
||||
const expected = 'reactivity';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle files without .md extension', () => {
|
||||
const input = 'docs/svelte/01-introduction';
|
||||
const expected = 'introduction';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle complex numbered prefixes', () => {
|
||||
const input = 'docs/svelte/99-advanced-topics.md';
|
||||
const expected = 'advanced-topics';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle files with multiple numbered prefixes', () => {
|
||||
const input = 'docs/svelte/01-02-nested-numbering.md';
|
||||
const expected = '02-nested-numbering';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle just a filename', () => {
|
||||
const input = '01-introduction.md';
|
||||
const expected = 'introduction';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle empty string', () => {
|
||||
const input = '';
|
||||
const expected = '';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle paths with no filename', () => {
|
||||
const input = 'docs/svelte/';
|
||||
const expected = '';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle files with hyphens but no numbers', () => {
|
||||
const input = 'docs/svelte/state-management.md';
|
||||
const expected = 'state-management';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle files with numbers in the middle', () => {
|
||||
const input = 'docs/svelte/svelte5-features.md';
|
||||
const expected = 'svelte5-features';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle tutorial paths', () => {
|
||||
const input = 'tutorial/01-introduction/01-hello-world.md';
|
||||
const expected = 'hello-world';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle SvelteKit paths', () => {
|
||||
const input = 'docs/kit/01-routing.md';
|
||||
const expected = 'routing';
|
||||
expect(extractTitleFromPath(input)).toBe(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('removeFrontmatter', () => {
|
||||
it('should remove valid frontmatter from content', () => {
|
||||
const input = `---
|
||||
title: Introduction
|
||||
description: Getting started guide
|
||||
---
|
||||
|
||||
# Introduction
|
||||
|
||||
This is the main content.`;
|
||||
const expected = `# Introduction
|
||||
|
||||
This is the main content.`;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle content without frontmatter', () => {
|
||||
const input = `# Introduction
|
||||
|
||||
This is content without frontmatter.`;
|
||||
const expected = `# Introduction
|
||||
|
||||
This is content without frontmatter.`;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle empty content', () => {
|
||||
const input = '';
|
||||
const expected = '';
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle malformed frontmatter (no closing delimiter)', () => {
|
||||
const input = `---
|
||||
title: Introduction
|
||||
This is malformed frontmatter without closing delimiter
|
||||
|
||||
# Content here`;
|
||||
const expected = input; // Should return original content unchanged
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle frontmatter with complex YAML', () => {
|
||||
const input = `---
|
||||
title: Complex Example
|
||||
tags:
|
||||
- svelte
|
||||
- tutorial
|
||||
metadata:
|
||||
author: John Doe
|
||||
date: 2024-01-15
|
||||
---
|
||||
|
||||
# Complex Example
|
||||
|
||||
Content with complex frontmatter.`;
|
||||
const expected = `# Complex Example
|
||||
|
||||
Content with complex frontmatter.`;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle content that starts with --- but is not frontmatter', () => {
|
||||
const input = `---
|
||||
This is not YAML frontmatter, just content that starts with ---`;
|
||||
const expected = input; // Should return original content unchanged
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle frontmatter with empty lines', () => {
|
||||
const input = `---
|
||||
title: Introduction
|
||||
|
||||
description: A guide
|
||||
---
|
||||
|
||||
# Content`;
|
||||
const expected = `# Content`;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should trim whitespace after removing frontmatter', () => {
|
||||
const input = `---
|
||||
title: Introduction
|
||||
---
|
||||
|
||||
|
||||
# Content with leading whitespace`;
|
||||
const expected = `# Content with leading whitespace`;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
|
||||
it('should handle frontmatter at the end of content', () => {
|
||||
const input = `---
|
||||
title: Only Frontmatter
|
||||
---`;
|
||||
const expected = ``;
|
||||
expect(removeFrontmatter(input)).toBe(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('integration tests', () => {
|
||||
it('should work together for typical documentation workflow', () => {
|
||||
// Simulate a typical path from tarball to display
|
||||
const tarball_path = 'svelte.dev-main/apps/svelte.dev/content/docs/svelte/01-introduction.md';
|
||||
|
||||
// Clean tarball path
|
||||
const cleaned_from_tarball = cleanTarballPath(tarball_path);
|
||||
expect(cleaned_from_tarball).toBe('apps/svelte.dev/content/docs/svelte/01-introduction.md');
|
||||
|
||||
// This would be stored in DB and later cleaned for display
|
||||
const cleaned_for_display = cleanDocumentationPath(cleaned_from_tarball);
|
||||
expect(cleaned_for_display).toBe('docs/svelte/01-introduction.md');
|
||||
|
||||
// Extract title for metadata
|
||||
const title = extractTitleFromPath(cleaned_from_tarball);
|
||||
expect(title).toBe('introduction');
|
||||
});
|
||||
|
||||
it('should handle SvelteKit paths through full workflow', () => {
|
||||
const tarball_path = 'svelte.dev-main/apps/svelte.dev/content/docs/kit/01-routing.md';
|
||||
|
||||
const cleaned_from_tarball = cleanTarballPath(tarball_path);
|
||||
expect(cleaned_from_tarball).toBe('apps/svelte.dev/content/docs/kit/01-routing.md');
|
||||
|
||||
const cleaned_for_display = cleanDocumentationPath(cleaned_from_tarball);
|
||||
expect(cleaned_for_display).toBe('docs/kit/01-routing.md');
|
||||
|
||||
const title = extractTitleFromPath(cleaned_from_tarball);
|
||||
expect(title).toBe('routing');
|
||||
});
|
||||
|
||||
it('should handle tutorial paths through full workflow', () => {
|
||||
const tarball_path =
|
||||
'svelte.dev-main/apps/svelte.dev/content/tutorial/01-introduction/01-hello-world.md';
|
||||
|
||||
const cleaned_from_tarball = cleanTarballPath(tarball_path);
|
||||
expect(cleaned_from_tarball).toBe(
|
||||
'apps/svelte.dev/content/tutorial/01-introduction/01-hello-world.md',
|
||||
);
|
||||
|
||||
const cleaned_for_display = cleanDocumentationPath(cleaned_from_tarball);
|
||||
expect(cleaned_for_display).toBe('tutorial/01-introduction/01-hello-world.md');
|
||||
|
||||
const title = extractTitleFromPath(cleaned_from_tarball);
|
||||
expect(title).toBe('hello-world');
|
||||
});
|
||||
|
||||
it('should handle content processing with frontmatter removal', () => {
|
||||
const content = `---
|
||||
title: Introduction
|
||||
---
|
||||
|
||||
# Introduction
|
||||
|
||||
This is the content.`;
|
||||
|
||||
const content_without_frontmatter = removeFrontmatter(content);
|
||||
expect(content_without_frontmatter).toBe(`# Introduction
|
||||
|
||||
This is the content.`);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,110 +0,0 @@
|
||||
/**
|
||||
* Unified path utilities for handling documentation paths
|
||||
*/
|
||||
|
||||
/**
|
||||
* Clean a path by removing the "apps/svelte.dev/content/" prefix
|
||||
* This is used to convert database paths to display paths
|
||||
*
|
||||
* @param path - The path to clean
|
||||
* @returns The cleaned path
|
||||
*/
|
||||
export function cleanDocumentationPath(path: string): string {
|
||||
const prefix = 'apps/svelte.dev/content/';
|
||||
if (path.startsWith(prefix)) {
|
||||
return path.substring(prefix.length);
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean a tarball path by removing the repository directory prefix (first segment)
|
||||
* This is used when processing files from GitHub tarballs
|
||||
*
|
||||
* @param path - The path to clean
|
||||
* @returns The cleaned path without the repo directory prefix
|
||||
*/
|
||||
export function cleanTarballPath(path: string): string {
|
||||
// Remove only the repo directory prefix (first segment)
|
||||
return path.split('/').slice(1).join('/');
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the title from a file path by removing prefixes and file extensions
|
||||
*
|
||||
* @param filePath - The file path to extract title from
|
||||
* @returns The extracted title
|
||||
*/
|
||||
export function extractTitleFromPath(filePath: string): string {
|
||||
if (!filePath) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const pathParts = filePath.split('/');
|
||||
const filename = pathParts[pathParts.length - 1];
|
||||
|
||||
// Handle empty filename (e.g., paths ending with '/')
|
||||
if (!filename) {
|
||||
return '';
|
||||
}
|
||||
|
||||
// Remove .md extension and numbered prefixes
|
||||
return filename.replace('.md', '').replace(/^\d+-/, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove frontmatter from markdown content using a tokenizer approach
|
||||
* Frontmatter is YAML metadata at the beginning of files between --- delimiters
|
||||
*
|
||||
* @param content - The markdown content that may contain frontmatter
|
||||
* @returns The content with frontmatter removed
|
||||
*/
|
||||
export function removeFrontmatter(content: string): string {
|
||||
if (!content || content.length === 0) {
|
||||
return content;
|
||||
}
|
||||
|
||||
// Check if content starts with frontmatter delimiter
|
||||
if (!content.startsWith('---\n')) {
|
||||
return content;
|
||||
}
|
||||
|
||||
let position = 4; // Start after the opening "---\n"
|
||||
let insideFrontmatter = true;
|
||||
let frontmatterEndOffset: number | null = null;
|
||||
|
||||
// Traverse the string character by character
|
||||
while (position < content.length && insideFrontmatter) {
|
||||
const char = content[position];
|
||||
|
||||
// Look for potential end of frontmatter: \n---
|
||||
if (char === '\n' && position + 3 < content.length) {
|
||||
const nextThree = content.substring(position + 1, position + 4);
|
||||
if (nextThree === '---') {
|
||||
// Check what comes after the closing ---
|
||||
const afterClosing = position + 4;
|
||||
|
||||
if (afterClosing >= content.length) {
|
||||
// End of string - this is valid frontmatter
|
||||
frontmatterEndOffset = content.length;
|
||||
insideFrontmatter = false;
|
||||
} else if (content[afterClosing] === '\n') {
|
||||
// Followed by newline - this is valid frontmatter
|
||||
frontmatterEndOffset = afterClosing + 1;
|
||||
insideFrontmatter = false;
|
||||
}
|
||||
// If followed by something else, it's not the end delimiter, continue searching
|
||||
}
|
||||
}
|
||||
|
||||
position++;
|
||||
}
|
||||
|
||||
// If we never found the end of frontmatter, it's malformed
|
||||
if (frontmatterEndOffset === null) {
|
||||
return content;
|
||||
}
|
||||
|
||||
// Return content after the frontmatter, trimmed
|
||||
return content.substring(frontmatterEndOffset).trim();
|
||||
}
|
||||
@@ -1,165 +0,0 @@
|
||||
export const SVELTE_5_PROMPT =
|
||||
'Always use Svelte 5 runes and Svelte 5 syntax. Runes do not need to be imported, they are globals. $state() runes are always declared using `let`, never with `const`. When passing a function to $derived, you must always use $derived.by(() => ...). Error boundaries can only catch errors during component rendering and at the top level of an $effect inside the error boundary. Error boundaries do not catch errors in onclick or other event handlers.';
|
||||
|
||||
export const DISTILLATION_PROMPT = `
|
||||
You are an expert in web development, specifically Svelte 5 and SvelteKit. Your task is to condense and distill the Svelte documentation into a concise format while preserving the most important information.
|
||||
Shorten the text information AS MUCH AS POSSIBLE while covering key concepts.
|
||||
|
||||
Focus on:
|
||||
1. Code examples with short explanations of how they work
|
||||
2. Key concepts and APIs with their usage patterns
|
||||
3. Important gotchas and best practices
|
||||
4. Patterns that developers commonly use
|
||||
|
||||
Remove:
|
||||
1. Redundant explanations
|
||||
2. Verbose content that can be simplified
|
||||
3. Marketing language
|
||||
4. Legacy or deprecated content
|
||||
5. Anything else that is not strictly necessary
|
||||
|
||||
Keep your output in markdown format. Preserve code blocks with their language annotations.
|
||||
Maintain headings but feel free to combine or restructure sections to improve clarity.
|
||||
|
||||
Make sure all code examples use Svelte 5 runes syntax ($state, $derived, $effect, etc.)
|
||||
|
||||
Keep the following Svelte 5 syntax rules in mind:
|
||||
* There is no colon (:) in event modifiers. You MUST use "onclick" instead of "on:click".
|
||||
* Runes do not need to be imported, they are globals.
|
||||
* $state() runes are always declared using let, never with const.
|
||||
* When passing a function to $derived, you must always use $derived.by(() => ...).
|
||||
* Error boundaries can only catch errors during component rendering and at the top level of an $effect inside the error boundary.
|
||||
* Error boundaries do not catch errors in onclick or other event handlers.
|
||||
|
||||
IMPORTANT: All code examples MUST come from the documentation verbatim, do NOT create new code examples. Do NOT modify existing code examples.
|
||||
IMPORTANT: Because of changes in Svelte 5 syntax, do not include content from your existing knowledge, you may only use knowledge from the documentation to condense.
|
||||
|
||||
Here is the documentation you must condense:
|
||||
|
||||
`;
|
||||
|
||||
export const SVELTE_DEVELOPER_PROMPT = `You are an expert in web development, specifically Svelte 5 and SvelteKit, with expert-level knowledge of Svelte 5, SvelteKit, and TypeScript.
|
||||
|
||||
## Core Expertise:
|
||||
|
||||
### Svelte 5 Runes & Reactivity
|
||||
- **$state**: Reactive state declaration (always use let, never const)
|
||||
- **$derived**: Computed values (always use $derived.by(() => ...) for functions)
|
||||
- **$effect**: Side effects and cleanup (runs after DOM updates)
|
||||
- **$props**: Component props with destructuring and defaults
|
||||
- **$bindable**: Two-way binding for props
|
||||
|
||||
### Critical Syntax Rules:
|
||||
${SVELTE_5_PROMPT}
|
||||
|
||||
### Additional Rules:
|
||||
- Props: let { count = 0, name } = $props()
|
||||
- Bindable: let { value = $bindable() } = $props()
|
||||
- Children: let { children } = $props()
|
||||
- Cleanup: $effect(() => { return () => cleanup() })
|
||||
- Context: setContext/getContext work with runes
|
||||
- Snippets: {#snippet name(params)} for reusable templates
|
||||
|
||||
### SvelteKit Essentials:
|
||||
- File-based routing with route groups and parameters
|
||||
- Load functions: +page.ts (universal) vs +page.server.ts (server-only)
|
||||
- Form actions in +page.server.ts with progressive enhancement
|
||||
- Layout nesting and data inheritance
|
||||
- Error and loading states with +error.svelte and loading UI
|
||||
|
||||
### TypeScript Integration:
|
||||
- Always use TypeScript for type safety
|
||||
- Properly type PageData, PageLoad, Actions, RequestHandler
|
||||
- Generic components with proper type inference
|
||||
- .svelte.ts for shared reactive state
|
||||
|
||||
## MCP Tool Usage Guide:
|
||||
|
||||
### Template Prompts (Efficient Documentation Injection):
|
||||
Use these for instant access to curated documentation sets:
|
||||
- **svelte-core**: Core Svelte 5 (introduction, runes, template syntax, styling)
|
||||
- **svelte-advanced**: Advanced Svelte 5 (special elements, runtime, misc)
|
||||
- **svelte-complete**: Complete Svelte 5 documentation
|
||||
- **sveltekit-core**: Core SvelteKit (getting started, core concepts)
|
||||
- **sveltekit-production**: Production SvelteKit (build/deploy, advanced, best practices)
|
||||
- **sveltekit-complete**: Complete SvelteKit documentation
|
||||
|
||||
### Resources Access:
|
||||
- **📦 Preset Resources**: Use svelte-llm://svelte-core, svelte-llm://svelte-advanced, svelte-llm://svelte-complete, svelte-llm://sveltekit-core, svelte-llm://sveltekit-production, svelte-llm://sveltekit-complete for curated documentation sets
|
||||
- **📄 Individual Docs**: Use svelte-llm://doc/[path] for specific documentation files
|
||||
- Access via list_resources or direct URI for browsing and reference
|
||||
|
||||
### When to use list_sections + get_documentation:
|
||||
- **Specific Topics**: When you need particular sections not covered by presets
|
||||
- **Custom Combinations**: When presets don't match the exact scope needed
|
||||
- **Deep Dives**: When you need detailed information on specific APIs
|
||||
- **Troubleshooting**: When investigating specific issues or edge cases
|
||||
|
||||
### Strategic Approach:
|
||||
1. **Start with Template Prompts**: Use template prompts (svelte-core, sveltekit-core, etc.) for immediate context injection
|
||||
2. **Browse via Resources**: Use preset resources for reading/reference during development
|
||||
3. **Supplement with Specific Docs**: Use list_sections + get_documentation only when presets don't cover your needs
|
||||
4. **Combine Efficiently**: Use multiple template prompts if you need both Svelte and SvelteKit context
|
||||
|
||||
### Documentation Fetching Priority:
|
||||
1. **Template Prompts First**: Always try relevant template prompts before individual sections
|
||||
2. **Preset Resources**: Use for browsing and reference
|
||||
3. **Individual Sections**: Only when specific content not in presets is needed
|
||||
4. **Multiple Sources**: Combine template prompts with specific sections as needed
|
||||
|
||||
## Best Practices:
|
||||
- Write production-ready TypeScript code
|
||||
- Include proper error handling and loading states
|
||||
- Consider accessibility (ARIA, keyboard navigation)
|
||||
- Optimize for performance (lazy loading, minimal reactivity)
|
||||
- Use semantic HTML and proper component composition
|
||||
- Implement proper cleanup in effects
|
||||
- Handle edge cases and provide fallbacks`;
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention, func-style
|
||||
export const createSvelteDeveloperPromptWithTask = (task?: string): string => {
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
const basePrompt = SVELTE_DEVELOPER_PROMPT;
|
||||
|
||||
if (!task) {
|
||||
return (
|
||||
basePrompt +
|
||||
`
|
||||
|
||||
## Your Approach:
|
||||
When helping with Svelte/SvelteKit development:
|
||||
1. **Use Template Prompts**: Start with relevant template prompts (svelte-core, sveltekit-core, etc.) for immediate context
|
||||
2. **Supplement as Needed**: Use list_sections + get_documentation only for content not covered by templates
|
||||
3. **Provide Complete Solutions**: Include working TypeScript code with proper types
|
||||
4. **Explain Trade-offs**: Discuss architectural decisions and alternatives
|
||||
5. **Optimize**: Suggest performance improvements and best practices`
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
basePrompt +
|
||||
`
|
||||
|
||||
## Current Task:
|
||||
${task}
|
||||
|
||||
## Task-Specific Approach:
|
||||
1. **Inject Relevant Context**: Use appropriate template prompts based on "${task.substring(0, 50)}...":
|
||||
- Component tasks: Use svelte-core for runes, template syntax
|
||||
- Advanced features: Use svelte-advanced for special elements, runtime
|
||||
- Full applications: Use svelte-complete + sveltekit-core/complete
|
||||
- Production apps: Use sveltekit-production for deployment, best practices
|
||||
2. **Supplement with Specific Docs**: Use list_sections + get_documentation only if templates don't cover specific needs
|
||||
3. **Design Architecture**:
|
||||
- Component structure and composition
|
||||
- State management approach
|
||||
- TypeScript types and interfaces
|
||||
- Error handling strategy
|
||||
4. **Implement Solution**:
|
||||
- Complete, working code
|
||||
- Proper types and error boundaries
|
||||
- Performance optimizations
|
||||
- Accessibility considerations
|
||||
5. **Explain Implementation**: Provide rationale for choices and discuss alternatives`
|
||||
);
|
||||
};
|
||||
@@ -1,8 +0,0 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import type { RequestHandler } from './$types';
|
||||
import { fetchRepositoryTarball } from '$lib/fetchMarkdown';
|
||||
|
||||
export const GET: RequestHandler = async () => {
|
||||
const tarball_buffer = await fetchRepositoryTarball('sveltejs', 'svelte.dev');
|
||||
return json({ data: tarball_buffer });
|
||||
};
|
||||
54
apps/mcp-remote/src/routes/webhooks/docs/+server.ts
Normal file
54
apps/mcp-remote/src/routes/webhooks/docs/+server.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
import { github_content_schema, github_webhook_schema } from '$lib/schemas/index.js';
|
||||
import * as v from 'valibot';
|
||||
|
||||
export async function POST({ request, fetch }) {
|
||||
const body = await request.json();
|
||||
// TODO add secret validation
|
||||
const validated_pull_request = v.safeParse(github_webhook_schema, body);
|
||||
if (!validated_pull_request.success) {
|
||||
return new Response('Invalid payload', { status: 400 });
|
||||
}
|
||||
|
||||
const { pull_request } = validated_pull_request.output;
|
||||
|
||||
if (!pull_request.merged) {
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
|
||||
const patch = await fetch(pull_request.patch_url);
|
||||
if (!patch.ok) {
|
||||
return new Response('Failed to fetch patch', { status: 500 });
|
||||
}
|
||||
|
||||
const patch_text = await patch.text();
|
||||
const files = [
|
||||
...patch_text.matchAll(
|
||||
/^diff --git\sa\/(?<file>.+?)\sb\/\1\n(?:(?<action>deleted|new)\sfile mode)?/gm,
|
||||
),
|
||||
].map((res) => ({ file: res.groups!.file, action: res.groups?.action ?? 'modified' }));
|
||||
|
||||
for (const file of files) {
|
||||
if (file.action === 'deleted') {
|
||||
// delete path from db
|
||||
continue;
|
||||
}
|
||||
const new_file_content = await fetch(
|
||||
`https://api.github.com/repos/sveltejs/svelte.dev/contents/${file.file}`,
|
||||
);
|
||||
if (!new_file_content.ok) {
|
||||
// push file in queue and try again later?
|
||||
continue;
|
||||
}
|
||||
const new_file_json = await new_file_content.json();
|
||||
const validated_content = v.safeParse(github_content_schema, new_file_json);
|
||||
if (!validated_content.success) {
|
||||
// push file in queue and try again later?
|
||||
continue;
|
||||
}
|
||||
const content = Buffer.from(validated_content.output.content, 'base64').toString('utf-8');
|
||||
// save content and distilled content in the db
|
||||
console.log({ content, file: file.file });
|
||||
}
|
||||
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
17
package.json
17
package.json
@@ -4,7 +4,6 @@
|
||||
"description": "The official Svelte MCP server implementation",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "pnpm --filter mcp-remote dev",
|
||||
"build": "pnpm -r run build",
|
||||
"check": "pnpm -r run check",
|
||||
"format": "prettier --write .",
|
||||
@@ -26,27 +25,13 @@
|
||||
"@eslint/compat": "^1.3.2",
|
||||
"@eslint/js": "^9.36.0",
|
||||
"@modelcontextprotocol/inspector": "^0.16.7",
|
||||
"@sveltejs/adapter-vercel": "^5.6.3",
|
||||
"@sveltejs/kit": "^2.22.0",
|
||||
"@sveltejs/vite-plugin-svelte": "^6.0.0",
|
||||
"@types/eslint-scope": "^8.3.2",
|
||||
"@types/estree": "^1.0.8",
|
||||
"@types/node": "^24.3.1",
|
||||
"@types/tar-stream": "^3.1.4",
|
||||
"@typescript-eslint/types": "^8.43.0",
|
||||
"dotenv": "^17.2.2",
|
||||
"drizzle-kit": "^0.30.2",
|
||||
"drizzle-orm": "^0.40.0",
|
||||
"eslint": "^9.36.0",
|
||||
"eslint-config-prettier": "^10.0.1",
|
||||
"eslint-plugin-import": "^2.32.0",
|
||||
"eslint-plugin-svelte": "^3.12.3",
|
||||
"globals": "^16.0.0",
|
||||
"minimatch": "^10.0.3",
|
||||
"prettier": "^3.4.2",
|
||||
"prettier-plugin-svelte": "^3.3.3",
|
||||
"svelte": "^5.0.0",
|
||||
"svelte-check": "^4.0.0",
|
||||
"tar-stream": "^3.1.7",
|
||||
"typescript": "^5.0.0",
|
||||
"typescript-eslint": "^8.44.1",
|
||||
"vitest": "^3.2.3"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { blob, integer, sqliteTable, text } from 'drizzle-orm/sqlite-core';
|
||||
import { integer, sqliteTable, text } from 'drizzle-orm/sqlite-core';
|
||||
import { float_32_array } from './utils.js';
|
||||
|
||||
/**
|
||||
@@ -51,7 +51,7 @@ export const distillation_jobs = sqliteTable('distillation_jobs', {
|
||||
|
||||
export const content = sqliteTable('content', {
|
||||
id: integer('id').primaryKey(),
|
||||
path: text('path').notNull().unique(),
|
||||
path: text('path').notNull(),
|
||||
filename: text('filename').notNull(),
|
||||
content: text('content').notNull(),
|
||||
size_bytes: integer('size_bytes').notNull(),
|
||||
@@ -67,7 +67,7 @@ export const content = sqliteTable('content', {
|
||||
|
||||
export const content_distilled = sqliteTable('content_distilled', {
|
||||
id: integer('id').primaryKey(),
|
||||
path: text('path').notNull().unique(),
|
||||
path: text('path').notNull(),
|
||||
filename: text('filename').notNull(),
|
||||
content: text('content').notNull(),
|
||||
size_bytes: integer('size_bytes').notNull(),
|
||||
@@ -80,17 +80,3 @@ export const content_distilled = sqliteTable('content_distilled', {
|
||||
.notNull()
|
||||
.$defaultFn(() => new Date()),
|
||||
});
|
||||
|
||||
export const cache = sqliteTable('cache', {
|
||||
id: integer('id').primaryKey(),
|
||||
cache_key: text('cache_key').notNull().unique(),
|
||||
data: blob('data', { mode: 'buffer' }).notNull(),
|
||||
size_bytes: integer('size_bytes').notNull(),
|
||||
expires_at: integer('expires_at', { mode: 'timestamp' }).notNull(),
|
||||
created_at: integer('created_at', { mode: 'timestamp' })
|
||||
.notNull()
|
||||
.$defaultFn(() => new Date()),
|
||||
updated_at: integer('updated_at', { mode: 'timestamp' })
|
||||
.notNull()
|
||||
.$defaultFn(() => new Date()),
|
||||
});
|
||||
|
||||
@@ -10,279 +10,292 @@ function run_autofixers_on_code(code: string, desired_svelte_version = 5) {
|
||||
return content;
|
||||
}
|
||||
|
||||
function with_possible_inits(title: string, fn: (args: { init: string }) => void) {
|
||||
describe.each([
|
||||
{ init: '$state' },
|
||||
{ init: '$state.raw' },
|
||||
{ init: '$derived' },
|
||||
{ init: '$derived.by' },
|
||||
])(title, fn);
|
||||
}
|
||||
|
||||
describe('add_autofixers_issues', () => {
|
||||
describe('assign_in_effect', () => {
|
||||
it(`should add suggestions when assigning to a stateful variable inside an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
});
|
||||
</script>`);
|
||||
with_possible_inits('($init)', ({ init }) => {
|
||||
it(`should add suggestions when assigning to a stateful variable inside an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = ${init}(0);
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
});
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add a suggestion for each variable assigned within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
const count2 = $state(0);
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
count2 = 44;
|
||||
});
|
||||
</script>`);
|
||||
it(`should add a suggestion for each variable assigned within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
const count2 = $state(0);
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
count2 = 44;
|
||||
});
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(2);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count2" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
it(`should not add a suggestion for variables that are not assigned within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
</script>
|
||||
|
||||
<button onclick={() => count = 43}>Increment</button>
|
||||
`);
|
||||
|
||||
expect(content.suggestions).not.toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
|
||||
it("should not add a suggestions for variables that are assigned within an effect but aren't stateful", () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = 0;
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(2);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count2" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
it(`should not add a suggestion for variables that are not assigned within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = ${init}(0);
|
||||
</script>
|
||||
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
});
|
||||
</script>`);
|
||||
<button onclick={() => count = 43}>Increment</button>
|
||||
`);
|
||||
|
||||
expect(content.suggestions).not.toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
expect(content.suggestions).not.toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add a suggestion for variables that are assigned within an effect with an update`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
let count = $state(0);
|
||||
|
||||
$effect(() => {
|
||||
count++;
|
||||
});
|
||||
</script>
|
||||
`);
|
||||
it("should not add a suggestions for variables that are assigned within an effect but aren't stateful", () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = 0;
|
||||
|
||||
$effect(() => {
|
||||
count = 43;
|
||||
});
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
expect(content.suggestions).not.toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add a suggestion for variables that are mutated within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
let count = $state({ value: 0 });
|
||||
|
||||
$effect(() => {
|
||||
count.value = 42;
|
||||
});
|
||||
</script>
|
||||
`);
|
||||
it(`should add a suggestion for variables that are assigned within an effect with an update`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
let count = ${init}(0);
|
||||
|
||||
$effect(() => {
|
||||
count++;
|
||||
});
|
||||
</script>
|
||||
`);
|
||||
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add a suggestion for variables that are mutated within an effect`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
let count = ${init}({ value: 0 });
|
||||
|
||||
$effect(() => {
|
||||
count.value = 42;
|
||||
});
|
||||
</script>
|
||||
`);
|
||||
|
||||
expect(content.suggestions).toContain(
|
||||
'The stateful variable "count" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.',
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe.each([{ method: 'set' }, { method: 'update' }])(
|
||||
'wrong_property_access_state ($method)',
|
||||
({ method }) => {
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with a literal init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
with_possible_inits('($init)', ({ init }) => {
|
||||
describe.each([{ method: 'set' }, { method: 'update' }])(
|
||||
'wrong_property_access_state ($method)',
|
||||
({ method }) => {
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with a literal init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
const count = ${init}(0);
|
||||
function update_count() {
|
||||
count.${method}(43);
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them.`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them.`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with an array init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with an array init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state([0]);
|
||||
const count = ${init}([0]);
|
||||
function update_count() {
|
||||
count.${method}([1]);
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them.`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them.`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable ($state({}))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable (${init}({}))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state({ value: 0 });
|
||||
const count = ${init}({ value: 0 });
|
||||
function update_count() {
|
||||
count.${method}({ value: 43 });
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable ($state(new Class()))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable (${init}(new Class()))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(new Class());
|
||||
const count = ${init}(new Class());
|
||||
function update_count() {
|
||||
count.${method}(new Class());
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable ($state(variable_name))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when using .${method}() on a stateful variable with conditional if it's not sure if the method could actually be present on the variable (${init}(variable_name))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const { init } = $props();
|
||||
const count = $state(init);
|
||||
const count = ${init}(init);
|
||||
function update_count() {
|
||||
count.${method}(43);
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should not add suggestions when using .${method} on a stateful variable if it's not a method call`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should not add suggestions when using .${method} on a stateful variable if it's not a method call`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state({});
|
||||
const count = ${init}({});
|
||||
function update_count() {
|
||||
console.log(count.${method});
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions).not.toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
expect(content.suggestions).not.toContain(
|
||||
`You are trying to update the stateful variable "count" using "${method}". stateful variables should be updated with a normal assignment/mutation, do not use methods to update them. However I can't verify if "count" is a state variable of an object or a class with a "${method}" method on it. Please verify that before updating the code to use a normal assignment`,
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
describe.each([{ property: '$' }])(
|
||||
'wrong_property_access_state property ($property)',
|
||||
async ({ property }) => {
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with a literal init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
describe.each([{ property: '$' }])(
|
||||
'wrong_property_access_state property ($property)',
|
||||
async ({ property }) => {
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with a literal init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(0);
|
||||
const count = ${init}(0);
|
||||
function read_count() {
|
||||
count.${property};
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them.`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them.`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with an array init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with an array init`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state([1]);
|
||||
const count = ${init}([1]);
|
||||
function read_count() {
|
||||
count.${property};
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them.`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them.`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable ($state({}))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable (${init}({}))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state({ value: 0 });
|
||||
const count = ${init}({ value: 0 });
|
||||
function read_count() {
|
||||
count.${property};
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable ($state(new Class()))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable (${init}(new Class()))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const count = $state(new Class());
|
||||
const count = ${init}(new Class());
|
||||
function read_count() {
|
||||
count.${property};
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable ($state(variable_name))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
it(`should add suggestions when reading .${property} on a stateful variable with conditional if it's not sure if the property could actually be present on the variable (${init}(variable_name))`, () => {
|
||||
const content = run_autofixers_on_code(`
|
||||
<script>
|
||||
const { init } = $props();
|
||||
const count = $state(init);
|
||||
const count = ${init}(init);
|
||||
function read_count() {
|
||||
count.${property};
|
||||
}
|
||||
</script>`);
|
||||
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
expect(content.suggestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(content.suggestions).toContain(
|
||||
`You are trying to read the stateful variable "count" using "${property}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them. However I can't verify if "count" is a state variable of an object or a class with a "${property}" property on it. Please verify that before updating the code to use a normal access`,
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe('imported_runes', () => {
|
||||
describe.each([{ source: 'svelte' }, { source: 'svelte/runes' }])(
|
||||
|
||||
@@ -39,7 +39,7 @@ function visitor(
|
||||
const init = definition.node.init;
|
||||
if (
|
||||
init?.type === 'CallExpression' &&
|
||||
state.parsed.is_rune(init, ['$state', '$state.raw'])
|
||||
state.parsed.is_rune(init, ['$state', '$state.raw', '$derived', '$derived.by'])
|
||||
) {
|
||||
state.output.suggestions.push(
|
||||
`The stateful variable "${id.name}" is assigned inside an $effect which is generally consider a malpractice. Consider using $derived if possible.`,
|
||||
|
||||
@@ -22,7 +22,7 @@ export const wrong_property_access_state: Autofixer = {
|
||||
const init = definition.node.init;
|
||||
if (
|
||||
init?.type === 'CallExpression' &&
|
||||
state.parsed.is_rune(init, ['$state', '$state.raw'])
|
||||
state.parsed.is_rune(init, ['$state', '$state.raw', '$derived', '$derived.by'])
|
||||
) {
|
||||
let suggestion = is_property
|
||||
? `You are trying to read the stateful variable "${id.name}" using "${node.property.name}". stateful variables should be read just by accessing them like normal variable, do not use properties to read them.`
|
||||
|
||||
@@ -4,6 +4,7 @@ export function list_sections(server: SvelteMcp) {
|
||||
server.resource(
|
||||
{
|
||||
name: 'list-sections',
|
||||
enabled: () => false,
|
||||
description:
|
||||
'The list of all the available Svelte 5 and SvelteKit documentation sections in a structured format.',
|
||||
uri: 'svelte://list-sections',
|
||||
|
||||
@@ -5,6 +5,7 @@ export function get_documentation(server: SvelteMcp) {
|
||||
server.tool(
|
||||
{
|
||||
name: 'get-documentation',
|
||||
enabled: () => false,
|
||||
description:
|
||||
'Retrieves full documentation content for Svelte 5 or SvelteKit sections. Supports flexible search by title (e.g., "$state", "routing") or file path (e.g., "docs/svelte/state.md"). Can accept a single section name or an array of sections. Before running this, make sure to analyze the users query, as well as the output from list_sections (which should be called first). Then ask for ALL relevant sections the user might require. For example, if the user asks to build anything interactive, you will need to fetch all relevant runes, and so on.',
|
||||
schema: v.object({
|
||||
|
||||
@@ -4,6 +4,7 @@ export function list_sections(server: SvelteMcp) {
|
||||
server.tool(
|
||||
{
|
||||
name: 'list-sections',
|
||||
enabled: () => false,
|
||||
description:
|
||||
'Lists all available Svelte 5 and SvelteKit documentation sections in a structured format. Returns sections as a list of "* title: [section_title], path: [file_path]" - you can use either the title or path when querying a specific section via the get_documentation tool. Always run list_sections first for any query related to Svelte development to discover available content.',
|
||||
},
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@sveltejs/mcp-server": "workspace:^",
|
||||
"@tmcp/transport-stdio": "^0.3.0",
|
||||
"@tmcp/transport-stdio": "^0.3.1",
|
||||
"@types/node": "^22.15.17",
|
||||
"publint": "^0.3.13",
|
||||
"tsdown": "^0.11.9",
|
||||
|
||||
908
pnpm-lock.yaml
generated
908
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user