CherryHQ-cherry-studio/docs/references/data/v2-migration-guide.md

# Migration V2 (Main Process)

Architecture for the new one-shot migration from the legacy Dexie + Redux Persist stores into the SQLite schema. This module owns orchestration, data access helpers, migrator plugins, and IPC entry points used by the renderer migration window.

## Version Upgrade Requirements

The v2 migration system enforces a **linear upgrade path** to ensure
data integrity:

```
v1.old  →  v1.last (≥1.9.0)  →  v2.0.0  →  v2.x
```

### Why a linear path?

v2.0.0 contains the one-shot data migration from Redux/Dexie to SQLite.
Supporting migration from every v1 version would create an O(n²) test
matrix. By requiring all users to be on the final v1 release first, the
migration code only needs to handle a single source data format.

### How it works

1. **VersionService** has been embedded since v1.7. It writes a
   `version.log` file to `{userData}/` on every launch where the
   version changes.
2. On v2 first launch, `v2MigrationGate.ts` reads `version.log` via
   `MigrationPaths.versionLogFile` (using the resolved userData path
   that accounts for v1 custom directories).
3. If the previous version is too old, missing, or if the user skipped
   v2.0.0, the gate shows an error dialog and quits.

### Blocking rules

| Scenario | Block reason | User action |
|----------|-------------|-------------|
| No `version.log` (v1 < 1.7 user) | `no_version_log` | Install v1.last, run once, then install v2.0.0 |
| Previous version < 1.9.0 | `v1_too_old` | Upgrade to v1.last first |
| Previous version is v1.x but current > v2.0.0 | `v2_gateway_skipped` | Install v2.0.0 first |

### Pre-release versions

v2.0.0 pre-releases (alpha/beta/rc) are treated as **before v2.0.0**
per semver ordering. They are allowed as migration targets from v1.last
(the gateway check coerces `currentVersion`, so `2.0.0-alpha` → `2.0.0`
passes). Pre-release to pre-release upgrades work because migration
status is `completed` after the first successful run.

The gateway is **strictly v2.0.0** — v2.0.x patches are blocked from
being a first migration target. This may be relaxed in a future release.

### Relationship with the auto-updater

The auto-updater (`AppUpdaterService`) controls which versions are
offered via OTA using `minCompatibleVersion` in the remote config. The
migration gate is a **separate safety net** for users who manually
download and install a version. Both systems enforce compatible upgrade
paths but operate independently.

## Directory Layout

```
src/main/data/migration/v2/
├── core/              # Engine + shared context
├── migrators/         # Domain-specific migrators and mappings
├── utils/             # Data source readers (Redux, Dexie, streaming JSON)
├── window/            # IPC handlers + migration window manager
└── index.ts           # Public exports for main process
```

## Core Contracts

- `core/MigrationEngine.ts` coordinates all migrators in order, surfaces progress to the UI, and marks status in `app_state.key = 'migration_v2_status'`. It will clear new-schema tables before running and abort on any validation failure.
- `core/MigrationPaths.ts` defines `MigrationPaths` (a frozen object of pre-computed paths) and `resolveMigrationPaths()` which detects v1 legacy userData directories from `~/.cherrystudio/config/config.json`. Called once at the migration gate entry, before engine initialization. All migration code uses these paths instead of `app.getPath()` — see the **Path safety** convention below.
- `core/MigrationContext.ts` builds the shared context passed to every migrator:
  - `sources`: `ConfigManager` (ElectronStore), `ReduxStateReader` (parsed Redux Persist data), `DexieFileReader` (JSON exports), `LegacyHomeConfigReader` (v1 `~/.cherrystudio/config/config.json` for the config-file migration path used by `BootConfigMigrator`)
  - `db`: current SQLite connection
  - `paths`: `MigrationPaths` — pre-computed filesystem paths; migrators that need file paths use `ctx.paths` instead of `app.getPath()`
  - `sharedData`: `Map` for passing cross-cutting info between migrators
  - `logger`: `loggerService` scoped to migration
- `@shared/data/migration/v2/types` defines stages, results, and validation stats used across main and renderer.

## Migrators

- Base contract: extend `migrators/BaseMigrator.ts` and implement:
  - `id`, `name`, `description`, `order` (lower runs first)
  - `prepare(ctx)`: dry-run checks, counts, and staging data; return `PrepareResult`
  - `execute(ctx)`: perform inserts/updates; manage your own transactions; report progress via `reportProgress`; self-check FK integrity of owned tables via `assertOwnedForeignKeys` (see Conventions → Foreign keys)
  - `validate(ctx)`: verify counts and integrity; return `ValidateResult` with stats (`sourceCount`, `targetCount`, `skippedCount`) and any `errors`
- Registration: list migrators (in order) in `migrators/migratorRegistry.ts` so the engine can sort and run them.
- Current migrators (see `migrators/README-<name>.md` for detailed documentation):
  - `PreferencesMigrator` (implemented): maps ElectronStore + Redux settings to the `preference` table using `mappings/PreferencesMappings.ts`.
  - `ChatMigrator` (implemented): migrates topics and messages from Dexie to SQLite. See [`README-ChatMigrator.md`](../../../src/main/data/migration/v2/migrators/README-ChatMigrator.md).
  - `BootConfigMigrator` (implemented, file-target): migrates early-boot settings into the file-based `bootConfigService` (`~/.cherrystudio/boot-config.json`) rather than a SQLite table. Reads from Redux (`disableHardwareAcceleration`) and from the v1 home config file (`~/.cherrystudio/config/config.json`'s `appDataPath` → `app.user_data_path`) via a `'configfile'` source kind. See [`README-BootConfigMigrator.md`](../../../src/main/data/migration/v2/migrators/README-BootConfigMigrator.md).
  - `AssistantMigrator`, `KnowledgeMigrator` (placeholders): scaffolding and TODO notes for future tables.
- Conventions:
  - All logging goes through `loggerService` with a migrator-specific context.
  - Use `MigrationContext.sources` instead of accessing raw files/stores directly.
  - Use `sharedData` to pass IDs or lookup tables between migrators (e.g., assistant -> chat references) instead of re-reading sources.
  - Stream large Dexie exports (`JsonStreamReader`) and batch inserts to avoid memory spikes.
  - **Foreign keys are OFF for the whole migration — do NOT toggle them per-migrator**: better-sqlite3 keeps a single persistent connection open for the whole process, so the engine sets `PRAGMA foreign_keys = OFF` **once** on that connection (in `MigrationDbService`) and it stays in effect for the entire migration — there is no per-transaction reconnection that could reset it. This lets bulk inserts carry not-yet-resolved references (self-referencing `message.parentId`, or cross-domain refs a later migrator resolves). Integrity is verified in two layers: (1) each migrator calls `this.assertOwnedForeignKeys(ctx.db, [...])` at the end of `execute()` for the tables it owns, giving early, well-attributed failures; (2) the engine runs a whole-database `PRAGMA foreign_key_check` after all migrators complete (`MigrationEngine.verifyForeignKeys`) as the final backstop.
    - **Self-check scope**: pass only tables whose FKs are fully resolved when *your* migrator finishes. **Exclude** refs a later migrator resolves — e.g. `assistant_knowledge_base.knowledgeBaseId` is written by `AssistantMigrator` but only becomes valid after `KnowledgeMigrator` remaps/prunes it, so `KnowledgeMigrator` self-checks that table, not `AssistantMigrator`. Dedicated file association tables (for example `chat_message_file_ref`) may be self-checked by the migrator that owns both the source rows and ref rows.
  - Count validation is mandatory; engine will fail the run if `targetCount < sourceCount - skippedCount` or if `ValidateResult.errors` is non-empty.
  - Keep migrations idempotent per run—engine clears target tables before it starts, but each migrator should tolerate retries within the same run.
  - **Path safety**: All filesystem paths MUST come from `ctx.paths` (the `MigrationPaths` object). NEVER call `app.getPath('userData')` or construct paths with `path.join` from scratch. Doing so bypasses the v1 legacy userData detection and may cause data loss for users with custom `appDataPath` configurations. If you need a path not yet in `MigrationPaths`, add it to the interface — do not inline it.

## Utilities

- `utils/ReduxStateReader.ts`: safe accessor for categorized Redux Persist data with dot-path lookup.
- `utils/DexieFileReader.ts`: reads exported Dexie JSON tables; can stream large tables.
- `utils/JsonStreamReader.ts`: streaming reader with batching, counting, and sampling helpers for very large arrays.
- `utils/LegacyHomeConfigReader.ts`: synchronously reads the v1 `~/.cherrystudio/config/config.json` file and normalizes its `appDataPath` field (both the legacy string shape and the current `{ executablePath, dataPath }[]` shape) into a `Record<executablePath, dataPath> | null`. Used exclusively by `BootConfigMigrator`'s `'configfile'` source.

## Window & IPC Integration

- `window/MigrationIpcHandler.ts` exposes IPC channels for the migration UI:
  - Receives Redux data and Dexie export path, starts the engine, and streams progress back to renderer.
  - Manages retry/cancel/restart/skip actions.
- `window/MigrationWindowManager.ts` creates the frameless migration window, handles lifecycle, and relaunch instructions after completion in production.

## Implementation Checklist for New Migrators

- [ ] Add mapping definitions (if needed) under `migrators/mappings/`.
- [ ] Implement `prepare/execute/validate` with explicit counts, batch inserts, and integrity checks.
- [ ] Wire progress updates through `reportProgress` so UI shows per-migrator progress.
- [ ] Register the migrator in `migrators/migratorRegistry.ts` with the correct `order`.
- [ ] Add any new target tables to `MigrationEngine.verifyAndClearNewTables` once those tables exist.
- [ ] Self-check FK integrity at the end of `execute()` via `this.assertOwnedForeignKeys(ctx.db, [...ownedTables])`, excluding cross-domain-deferred refs and shared polymorphic tables (see Conventions → Foreign keys). Do NOT toggle `PRAGMA foreign_keys` yourself — the engine keeps it OFF for the whole migration.
- [ ] Include detailed comments for maintainability (file-level, function-level, logic blocks).
- [ ] **Create/update `migrators/README-<MigratorName>.md`** with detailed documentation including:
  - Data sources and target tables
  - Key transformations
  - Field mappings (source → target)
  - Dropped fields and rationale
  - Code quality notes

## Order-Key Stamping in Migrators

Legacy Redux/Dexie → SQLite migrators for sortable resources must produce `order_key` values for every row they insert. The v2 migrator layer owns a pair of **pure functions** under `src/main/data/migration/v2/utils/orderKey.ts` that handle this without touching the DB — they take a pre-flattened array and return the same rows with `orderKey` attached.

| Helper | Shape | Use for |
|---|---|---|
| `assignOrderKeysInSequence(rows)` | Returns `rows` with one monotonically increasing `orderKey` per row. | Whole-table ordering (e.g. `mcp_server`, `user_provider`, `miniapp`). |
| `assignOrderKeysByScope(rows, getScope)` | Groups rows by the scope key, stamps each bucket independently (independent key spaces per bucket). | Partitioned tables (e.g. `topic.groupId`, `user_model.providerId`, `group.entityType`). |

**Pattern — flatten first, stamp last:** keep `transform*` functions pure (no `index` parameter, no `sortOrder` argument); flatten the legacy source into an array, then stamp keys onto the whole array:

```typescript
import { assignOrderKeysByScope, assignOrderKeysInSequence } from '@data/migration/v2/utils/orderKey'

// Before — each transform took an index and emitted a sortOrder
const rows = legacyServers.map((src, i) => transformMcpServer(src, i).row)

// After — transforms are pure; keys are assigned after the flatten
const rows = legacyServers.map((src) => transformMcpServerV2(src).row)
const stamped = assignOrderKeysInSequence(rows)
await tx.insert(mcpServerTable).values(stamped)

// Partitioned example — each providerId becomes its own independent key space
const stamped = assignOrderKeysByScope(userModels, (m) => m.providerId)
```

**Import rule — never reach for `fractional-indexing` directly:** the migrator helpers delegate to `generateOrderKeySequence` exported from `src/main/data/services/utils/orderKey.ts`, which is the **single** sanctioned integration point for the library. Migrator code, migration scripts, and drizzle custom-migration callbacks all re-import from that service-layer wrapper. This keeps the library boundary auditable and leaves a single place to change the character set or swap implementations.

For the runtime counterparts (`insertWithOrderKey` / `insertManyWithOrderKey` / `applyMoves` / `resetOrder`) used outside the migration window, see [Reorder Guide — Server-Side Service Helpers](./data-ordering-guide.md#3-server-side-service-helpers).