Data Layer

Data Types

Canonically defined in packages/schema/src/ (@arkaik/schema) and re-exported through lib/data/types.ts:

Node

interface Node {
  id: string;
  project_id: string;
  species: SpeciesId;
  title: string;
  description?: string;
  status: StatusId;
  platforms: PlatformId[];
  metadata?: NodeMetadata;
}

NodeMetadata

Declared in packages/schema/src/bundle.ts (NodeMetadata). Most fields are species-scoped: the type is one open bag rather than a union, so a field's audience column matters as much as its type.

Unknown metadata keys are preserved (catchall) — the forward-compatibility rule of the format.

FlowPlaylist structure:

type PlaylistEntry =
  | { type: "view"; view_id: string }
  | { type: "flow"; flow_id: string }
  | { type: "condition"; label: string; if_true: PlaylistEntry[]; if_false: PlaylistEntry[] }
  | { type: "junction"; label: string; cases: JunctionCase[] };

interface JunctionCase {
  label: string;
  entries: PlaylistEntry[];
}

interface FlowPlaylist {
  entries: PlaylistEntry[];
}

Edge

interface Edge {
  id: string;
  project_id: string;
  source_id: string;
  target_id: string;
  edge_type: EdgeTypeId;
  metadata?: Record<string, unknown>;
}

Project

interface Project {
  id: string;
  title: string;
  description?: string;
  version?: string;    // Current version label of the mapped product (Level 1)
  root_node_id?: string; // Optional node id used as the canvas anchor
  metadata?: ProjectMetadata; // Optional project-level UI preferences
  created_at: string;  // ISO 8601
  updated_at: string;  // ISO 8601
  archived_at?: string | null; // ISO 8601 when archived
}

interface ProjectMetadata extends Record<string, unknown> {
  view_card_variant?: "compact" | "large"; // Deprecated; superseded by map_display, no longer read
  maps?: MapDefinition[]; // Stored map definitions (spec/maps.md § Storage)
  map_display?: Record<string, MapDisplayOptions>; // Per-map card rendering, keyed by map id
  products?: ProductDefinition[]; // Product definitions (spec/bundle-format.md § Products)
}

ProjectMetadata is a catchall too, and one spec'd key rides through it rather than being typed: ref_policy, the opt-in ref → status promotion map read by computeRefPromotions (spec/bundle-format.md § References). Absent it, nothing is promoted.

ProjectBundle

interface ProjectBundle {
  schema_version?: number;   // Absent means 1 (spec/bundle-format.md)
  project: Project;
  nodes: Node[];
  edges: Edge[];
  journal?: JournalEvent[];  // Level 2 embedded interchange projection (spec/journal.md)
}

A ProjectBundle is the unit of storage and export — one project with all its nodes and edges.

project.root_node_id (when present) points to the node the Journey map anchors on (lib/utils/journey-graph.ts); a scoped map's own root_node_id overrides it. If it is missing, the canvas falls back to inferred roots (nodes without compose parents).

DataProvider Interface

Declared in lib/data/data-provider.ts — read the signatures there. All data access goes through it. What follows is a map of the surface plus the reasoning behind the three shapes that surprise people; deliberately not a transcription, because the interface has been redesigned once already and a second copy of the signatures is exactly the thing that rots.

• listProjects() returns ProjectSummary[], not bundles. A summary is { project, nodeCount, edgeCount, hosted, seed? }. It used to return full ProjectBundles, which meant the projects page read every node, edge and journal event of every project just to render titles and counts — merely wasteful against IndexedDB, untenable against a server.
• Every mutator takes projectId explicitly, including the ones whose subject id would seem to be enough. The local provider can get away without it by scanning IndexedDB for the project holding a node id; a remote provider cannot scan, and the routing provider below could not even tell which backend to ask. It is free at the call sites — the hooks are already built as useNodes(projectId).
• applyMutations(projectId, ops) is the atomic batch: all ops commit, or none do. The single-op methods cannot express "create this node and this edge together", which forces callers into a create-then-create sequence with a hand-rolled rollback when the second half fails. The local provider runs one IndexedDB transaction; the remote provider sends one request. MutationOp comes from @arkaik/schema.
• applyMutations answers with more than the graph. Its MutationResult is { nodes, edges, version?, events? }: the whole graph after the write, plus — when the backend knows them — the server's strong version after the write and the journal events the write appended, in append order. Both exist for the query cache (§ Hooks below), which writes the result straight into its project entry instead of re-reading the project, guards that write with version, and extends its cached journal with events instead of re-downloading it.

archiveProject performs a soft delete. Archived projects remain in storage but are excluded by default from listProjects().

Providers

getProvider() (lib/data/provider-registry.ts) is the one seam every hook and component reads through; setProvider() swaps it (tests, and the read-only repo-bundle viewer of rfcs/arkaik-dev.md). Its default is not the local provider alone but a router over three implementations of the same interface:

The routing rule is the id namespace, not a cache (lib/data/routing-provider.ts): hosted ids are minted server-side and the import path refuses to give a local project one, so routing is a total function of the id with nothing to populate, invalidate, or get wrong offline. listProjects() is the one call that spans all three — it always leads with the seed, adds the account's hosted projects when /api/auth/status says there is an account, and degrades to the local list rather than blanking the page if that request fails.

The journal projection. getJournal(projectId, { types }) and readJournal's types narrow a journal read to a set of event types. The remote provider sends them as ?types=a,b; the local and seed providers apply the same filter in memory (lib/data/journal-projection.ts) — nothing to save there, but every backend then answers the same question, so a page can ask for its types without knowing where the project lives. [], null and undefined all mean the whole journal, in the cache key, in the filter and in the server's parser alike. Who asks for what:

A page may only project types it can render a truthful empty state for: "no journal yet" stops being sayable once the events that would contradict it are no longer read, which is why the changelog now says "No releases tagged yet, and nothing shipped since."

Conditional reads. Two optional methods sit beside the plain ones: readProject(id, { etag, signal }) and readJournal(projectId, { etag, types, signal }), each answering { status: "fresh", value, etag, version? }, { status: "not-modified" } or { status: "missing" }. The remote provider sends the caller's validator as If-None-Match and turns the server's 304 into not-modified — no body travels, because the one copy of a read lives in the query cache above it, not in the provider. They are optional because only a backend with a server validator has any use for them, and the routing provider owns the fallback: it forwards to a backend that implements them and otherwise wraps that backend's getProject/getJournal as a fresh answer with etag: null, which is exactly what an unconditional read is. Since getProvider() always answers with the router, a caller may call these for any project id. A 404 reads as missing; a 401 or a 500 still throws, so an auth failure can never be mistaken for an empty project. The read contract they speak to — which validator each route emits and which writes move it — is spec/services.md § Hosted Graph Projects → Read contract.

Signed out, or with services unconfigured, nothing reaches the network and the app behaves exactly as it did before hosted projects existed.

What each fills into applyMutations' result: the remote provider forwards the version and events the mutations route already returned; local and seed return the events their own runOps derived and appended (the same toJournalEvents(outcome.eventInputs) the journal row received) and no version — Dexie transactions and the in-memory sandbox serialize, so their results resolve in commit order and there is nothing to guard against. The routing provider passes the result through untouched.

Local Provider

Implemented in lib/data/local-provider.ts.

• Backend: IndexedDB via Dexie (lib/data/db.ts, database arkaik) — three tables: projects (one row per project: the bundle snapshot minus its journal), journals (per-project event arrays), meta (bookkeeping)
• Writes: Row-level per project — a mutation to project A rewrites only A's row
• Dual-write: Every graph mutation patches the snapshot and appends the derived journal events (lib/data/emit-events.ts, actor arkaik-app) in the same Dexie transaction; saveProject/importProject/archiveProject deliberately do not emit
• Notifications: subscribeToMutations(cb) fires per affected project after the transaction commits (consumed by the Synk SyncManager)
• Cascade: deleteNode also removes all edges referencing that node (no separate edge.removed events — implied by node.deleted)
• Legacy migration: on first open, any old arkaik:store localStorage payload is imported once (running migrateBundle per bundle) and kept as a passive backup
• Normalization: legacy structural fields are stripped and playlists hydrated via the explicit migration chain in lib/data/migrate.ts (schema_version-aware)

Import / Export

Utilities in lib/utils/export.ts:

• exportToJson(bundle) — Serializes a ProjectBundle to formatted JSON
• downloadJson(bundle) — Triggers browser download as {project-title-slug}-{projectId}.json and returns export diagnostics (filename, bytes, warning)
• exportProject(id) / importProject(bundle) — Delegate to getProvider(), so they follow the same routing as every other call
• importProjectFromFile(file) — Parses and validates JSON file content, normalizes timestamps, and imports via provider

downloadJson(bundle) applies a soft warning when the serialized bundle is larger than 4 MB. The warning is intended for UX guidance only and does not block download.

When importing, if the incoming project ID already exists locally, a new project ID is generated and all project_id references in nodes and edges are rewritten to the new ID before saving.

When importing JSON, project.root_node_id is optional. If provided, it must reference an existing node ID in nodes or the import fails validation.

Public Schema Contract

Arkaik now publishes a machine-readable schema and example bundle for import/export alignment and LLM prompt tooling:

These assets are generated from the canonical zod source in packages/schema (npm run generate, drift-checked in CI) and help external tooling generate importable bundles.

Hooks

Hooks in lib/hooks/ are thin bindings over a query cache (below); their return shapes are frozen:

The node panel's History section reads the journal itself (useJournal(projectId) inside NodeDetailPanel, keyed by the route id from useProjectId() — never node.project_id, which on a hosted project is the imported bundle's own id rather than the prj_… route id — and mounted when a page passes history to PageShell), so only the Changelog, Design, Decisions, History and Overview pages still request the journal at page level — the first three as projections of the event types they render, the last two whole — the maps, Library, Delivery and Acceptances no longer request it at page level (their node panels read it lazily on first open and share one cached entry), and the Overview no longer holds its first paint for it.

reload() is the retry behind every PageError on a project surface: it re-runs the read and resolves when it settles. A page that fans one retry into several reload() calls is re-running one query, so each call joins the fetch already in flight rather than cancelling it. loading is true whenever the data has not been read yet — never false with an empty list before a read — and a retry after an error with no data looks like a load again. error is null exactly when absent; a failed background refetch over data already on screen is not reported (the surface keeps what it has).

The Journey map (components/maps/JourneyMap.tsx) uses useProject for root-node anchoring and project-level card-style preferences, and still manages expandedFlows as local state.

The query cache

Every project surface used to run its own reads on mount, so one navigation cost up to six provider reads of the same project. The hooks now observe a TanStack Query cache — one QueryClient per browser (lib/data/query-client.ts), handed to React by components/query/QueryProvider.tsx in the root layout — described entirely in lib/data/project-queries.ts, which is the cache's only writer. Components never touch the client; the hooks are the binding layer, and everything else goes through a seam.

• Freshness. A project entry is fresh for 30 s (navigation inside the window is a cache hit with no request; beyond it the cached data paints immediately and a background refetch runs), survives 30 min unobserved, retries once and never on a 4xx. Refetch on window focus is on, listening to window focus as well as visibilitychange. The listing is fresh for 60 s. version guards write-backs; etag is the read validator the next refetch sends as If-None-Match — stored from the server's answer, and dropped to null by every write-back, since a written-to entry no longer matches any server ETag.
• Revalidation. Each read queryFn reads its own previous entry through the QueryFunctionContext's client, sends that entry's etag, and on not-modified returns that same entry object — identity, so structural sharing short-circuits and nothing downstream renders. A background revalidation of an unchanged hosted project is therefore one small request and zero renders. TanStack's signal is forwarded to fetch, so the cancelQueries every write-back opens with actually tears the request down. Hosted entries poll: refetchInterval: 60_000, not in the background, because a hosted project is written by agents, the CLI and the GitHub App while a tab is open and the remote provider has no mutation bus to hear it. Local and seed projects have one (or no other writer), and never poll.
• Write-back. Every hook mutator goes through provider.applyMutations and writes its MutationResult back: cancel any in-flight read of the entry (a refetch that started before the mutation must not land after it), then adopt nodes and edges under the version guard (a result whose version is older than the entry's is dropped — two hosted writes are routinely in flight together), then cancel any in-flight read of the journal projections (for the same reason) and append the returned events to every projection that admits them, or mark the projections stale without refetching when the backend returned none, then mark the listing stale. A cancelled read that had no data to revert to is re-issued for any mounted observer, so a first load overlapping a write never stays loading. syncEdges carries the result's version so its edge-only write-back keeps the same guard. updateProject re-reads the bundle straight from the provider (not through the entry: a write-back's cancel would hand a cache fetch the reverted snapshot) — conditionally, so on a hosted project a not-modified answer is served from the cached entry as a plain lookup rather than a second download — then saves and replaces the entry. Structural sharing is on and the selectors are module-level, so an unchanged refetch keeps project.project, nodes and edges identities and nothing downstream re-renders or re-lays out.
• Seams. Writers that bypass the hooks call invalidateProject(id) (bundle and journal, refetched where mounted — the raw-bundle save) or invalidateProjects() (the listing, marked stale for its next mount — import, archive, the projects page's create/import/move, and useAuthStatus when it resolves signed-in). Both read the client through getQueryClient(), so they are harmless on the server.
• The local bus. QueryProvider subscribes to the local provider's subscribeToMutations and invalidates ["project", id] and ["projects"] on every local write, so an import or a restore refreshes whatever is on screen without a seam. It fires for the hooks' own writes too; those cancel the bundle and journal entries before writing back, so the reads the bus started are ignored rather than adopted (Dexie has no abort — the reads themselves still complete). Seed and remote have no bus.

The suite is npm run test:project-queries (tests/data/project-queries.test.js), against the real TanStack core with the providers stubbed. The provider side of the same contract — conditional reads, the routing fallback, ?types= on the URL — is npm run test:provider, and the server's half of it is npm run test:graph-etag (pure, no database) and npm run test:graph (against a migrated Postgres).

Node Editing Flow

The NodeDetailPanel is the primary UI for editing nodes. The mutation path:

NodeDetailPanel (title, description, platforms, metadata)
  → useNodes.updateNode(id, patch)
    → getProvider().updateNode(projectId, id, patch)
      → local: IndexedDB (Dexie) | remote: POST /api/graph/projects/{id}/mutations | seed: per-tab memory

The hook closes over the projectId it was constructed with, which is why the UI never passes one and the provider always gets one.

Views store editable per-platform statuses in node.metadata.platformStatuses. When legacy data does not have that field yet, the UI derives platform statuses from node.status + node.platforms and writes the richer metadata shape back on the next edit.

Flows do not expose an editable rollup status in UI. Flow cards and panel gauges compute status from descendant views in lib/utils/journey-graph.ts and components/panels/NodeDetailPanel.tsx.

Flow playlist edits (metadata.playlist.entries) also originate from NodeDetailPanel via components/panels/PlaylistEditor.tsx. All playlist mutations use useNodes.updateNode, and provider-side validation blocks circular flow references before persistence.

What the seam bought

The DataProvider interface abstracts storage so the backend can change without touching hooks or UI. That has now been cashed in twice:

1. The local provider moved from localStorage to IndexedDB (Dexie — lib/data/db.ts) with the localProvider export name kept, so hooks and UI were unchanged.
2. A second backend arrived and the app did not notice. remote-provider.ts is a DataProvider over the hosted graph API, and routing-provider.ts dispatches to it by id; a third, seed-provider.ts, backs the public self-map from memory. No hook or component knows which one answered.

Where the source of truth lives depends on the project, and this is a deliberate crossing. For a local-first project the browser is still the system of record, and Publik and Synk remain what they always were — share and one-way backup, not providers (spec/services.md). For a hosted project the server is the system of record: it applies mutations under a row lock and the GitHub App promotes acceptance statuses from pull-request events with no browser involved. That reverses boundary 1 of the services spec, and the spec says so explicitly rather than quietly — read the decision record in spec/services.md § "Boundary 1 no longer holds for hosted projects" before designing anything on top of it, including its stated cost: hosted projects do not work offline, local-first ones still do and remain the default.

Every server-side mutation still passes the same validateBundle the CLI and MCP server use, in the same transaction, and is refused whole on any error; a hosted project can be exported and re-imported as a local one at any time.

Storage layout (local provider): a projects table keyed by id holds one row per project (the bundle snapshot minus its journal), so a mutation to project A rewrites only project A's row — not the whole store as the previous localStorage backend did. The embedded journal lives in its own journals table (keyed by projectId), leaving room for a future app-side journal append that need not rewrite the graph snapshot. On first load, any legacy arkaik:store localStorage payload is imported once into IndexedDB (running migrateBundle per bundle) and the source payload is kept as a passive backup.

To add a new provider: implement the DataProvider interface and inject it via setProvider() (lib/data/provider-registry.ts) — the seam every hook already reads through.

Seed Data

seed/pebbles.json contains an example project ("Pebbles") exercising all six species, persisted compose edges for structure, and playlist-driven flow ordering. seed/arkaik-self-map.json is Arkaik's own map, served read-write-in-memory as the built-in public project by seed-provider.ts. Both are validated in CI (npm run validate:seeds).