# Knowledge Schema Notes (V2) This document records the current V2 knowledge target schema, migration constraints, and temporary scope boundaries. ## Scope Clarification - `video` items are out of scope for V2 knowledge data migration and should be skipped. - `memory` items belong to the memory module, not the knowledge module, and should be skipped in knowledge migration. ## Current Target Schema ### `knowledge_base` - Persisted columns: - `id` - `name` - `groupId` - `dimensions` - `embeddingModelId` - `status` - `error` - `rerankModelId` - `fileProcessorId` - `chunkSize` - `chunkOverlap` - `threshold` - `documentCount` - `searchMode` - `hybridAlpha` - `createdAt` - `updatedAt` ### `knowledge_item` - Persisted columns: - `id` - `baseId` - `groupId` - `type` - `data` - `status` - `error` - `createdAt` - `updatedAt` - New app-created knowledge items use ordered UUID generation for `id`. ## Fields Removed From The V2 SQLite Schema - `video` is not a target `knowledge_item.type`. - `memory` is not a target `knowledge_item.type`. - `sitemap` is not a target `knowledge_item.type`; legacy sitemap entries are migrated as `url` items. - Legacy runtime-only item fields are not stored as standalone SQLite columns: - `uniqueId` - `uniqueIds` - `processingProgress` - `retryCount` - `isPreprocessed` - `remark` is not part of the V2 SQLite schema. - `sourceUrl` is not a standalone `knowledge_item` column: - for notes, it may exist inside `data.sourceUrl` - for url items, the URL is stored inside the typed `data` payload - Official v1 legacy exports do not contain `groupId`. ## `groupId` Semantics - `knowledge_item` is modeled as a flat same-base item collection. - `groupId` is an optional stable grouping key inside one knowledge base. - Typical usage: items from the same imported source/container - Examples: one directory import, one URL collection - When one item is the logical container/owner of a group, downstream items use `groupId = containerItem.id` - The schema enforces same-base ownership: - `(baseId, groupId)` must reference `(baseId, id)` in `knowledge_item` - deleting the owner cascades to grouped members - Current runtime read flows use: - `GET /knowledge-bases/:id/items` for flat item listing - optional query filters: `type`, `groupId` - Current runtime write workflows use `KnowledgeService` IPC, not DataApi endpoints: - add items: normalize caller-friendly inputs, create SQLite rows, and enqueue prepare/index tasks - delete items: interrupt runtime work, delete vectors, then delete SQLite roots - reindex items: interrupt runtime work, delete old vectors, rebuild expanded children when needed, then enqueue indexing - search and chunk mutation: execute against the per-base vector store through runtime IPC - DataApi remains limited to SQLite-backed reads and knowledge base metadata PATCH. - Migration from official v1 data does not preserve or infer grouping metadata: - official v1 exports are flat - migrated items are inserted with `groupId = null` ## Current `type` / `data` Integrity Boundary - `knowledge_item.type` and `knowledge_item.data` are intended to stay aligned by controlled UI flows. - In the current V2 scope, knowledge item create/edit operations are expected to come from strongly associated UI forms or controlled write paths for each item type. - The current implementation does not add an extra DB-level cross-structure constraint that re-validates `data` against the stored `type` on every write. - At the DataApi/service layer: - create flows still rely on controlled write paths to keep `type` and `data` aligned - update flows re-validate `data` against the stored `type` before persisting changes - Downstream knowledge code may therefore treat the stored `type` + `data` pair as a trusted contract produced by the app's controlled write path. - If future write paths are added outside the current controlled UI flow, such as import tools, scripts, sync jobs, or public/external APIs, this assumption must be revisited and explicit boundary validation should be added at that time. ## Current Non-Goals - This phase does not reconstruct hierarchy from legacy v1 exports. - This phase does not infer directory child relationships during migration. - This phase does not introduce a first-class `knowledge_group` table. - This phase does not preserve temporary processing lifecycle states beyond the `uniqueId`-based status rule below. - This phase does not migrate `video` or `memory` into V2 knowledge tables. ## `dimensions` Resolution Rule - `dimensions` is treated as a required field for target V2 `knowledge_base`. - Migration does not trust legacy Redux `dimensions` as the source of truth. - Migration must resolve `dimensions` from the legacy vector database by inspecting: - the per-base legacy vector DB file - the `vectors` table - a non-null vector blob whose byte length can be converted to a positive dimension count - Resolution is considered failed when the legacy vector DB is missing, empty, invalid, or its vector blob length cannot be parsed into a valid positive dimension count. - When resolution fails, the knowledge base is considered unusable in V2 migration: - skip the entire base - skip all items under that base - record a warning for diagnostics - Migration does not apply fallback or auto-fix for unresolved `dimensions`. ## Item Status Migration Rule - Legacy `processingStatus` is treated as runtime state and is not used as the migration source of truth. - Migration infers target V2 `knowledge_item.status` from legacy `uniqueId`: - non-empty `uniqueId` -> `completed` - otherwise -> `idle` - Temporary legacy states such as in-progress or failed processing are not preserved as V2 status during migration. ## Runtime Status Boundary - `knowledge_item.status` and `knowledge_item.error` remain part of the official V2 business schema. - The runtime queue implementation is not part of the schema contract: - no separate task table - no persisted queue record - no persisted task run id - Runtime currently uses an in-memory `p-queue` based pipeline in `KnowledgeRuntimeService`. - The schema-level `status` set is: - `idle` - `preparing` - `processing` - `reading` - `embedding` - `completed` - `failed` - `deleting` - Current runtime writes: - `preparing` while a `directory` root or nested directory is being expanded - `reading` while a leaf item is reading source documents - `embedding` while a leaf item is embedding / writing vectors - `processing` while a container has active descendants but is not itself expanding - `completed` after successful leaf indexing, or when a container has no active children - `failed` on runtime failure, interrupt cleanup failure, or shutdown interruption - `fileProcessorId` is persisted in base config, but it does not participate in runtime execution yet. - In other words: - queue structure is implementation detail - `status` is business lifecycle and coarse runtime progress - container status is reconciled from its own status and child item statuses - these concerns must not be conflated ## Current Runtime Consumption Notes - Runtime entrypoint: - `src/main/services/knowledge/runtime/KnowledgeRuntimeService.ts` - Reader dispatch code still exists for stored `knowledge_item.type` values: - `file` -> file reader by extension - `url` -> fetch markdown through Jina Reader - `note` -> inline note content - `directory` -> currently treated as a container placeholder and returns no documents - `sitemap` is no longer a valid persisted V2 `knowledge_item.type`. Legacy v1 sitemap items are mapped to `url` during migration and indexed through the URL path. - Runtime add flow accepts new item payloads: - leaf payloads create `knowledge_item` rows and enqueue `index-leaf` - `directory` payloads create root rows and enqueue `prepare-root` - `prepare-root` expands a directory owner inside the runtime queue, creates child rows, and enqueues concrete leaf children as `index-leaf`. - Callers must not create user-supplied nested `directory` items under another item. Nested directory rows may still be created internally by directory expansion to preserve filesystem hierarchy. - Runtime embedding model resolution currently expects `knowledge_base.embeddingModelId` in `providerId::modelId` format and only supports `ollama` as the active provider. ## Implementation Status - `video` and `memory` items are skipped during migration. - The target schema uses optional `groupId`, but migration from official v1 data still writes it as `null`. - The current DataApi contract exposes flat item read/listing only; write operations go through runtime orchestration. - Group ownership is represented implicitly by `groupId = ownerItem.id`; there is no standalone group table in the current phase. - `dimensions` resolution failure skips the entire base and all nested items, with warnings recorded in migration output. - Knowledge item status migration uses `uniqueId` instead of `processingStatus`. - The current runtime service is `KnowledgeRuntimeService`, not the old `KnowledgeService` name used in earlier notes. - Current runtime queue behavior is a single in-memory `PQueue({ concurrency: 5 })` shared across knowledge bases; there is no per-base serial queue yet. - Current runtime queue entries are `prepare-root` and `index-leaf`; preparation and leaf indexing share interrupt / wait / shutdown cleanup semantics.