DTIF parser guide

The canonical DTIF parser powers validation, normalisation, graph construction, and resolution across the toolchain. This guide covers installation, configuration, the high-level helper APIs, and the internals you need to integrate the parser into tooling and CI pipelines.

Installation

The parser is published as @lapidist/dtif-parser. Install it alongside the DTIF schema and validator packages:

npm install @lapidist/dtif-parser

The package ships ESM output, TypeScript declarations, and a dtif-parse CLI binary.

Quick start

Parse sessions

Create a new parse session to reuse caches and configuration across documents:

import { createSession } from '@lapidist/dtif-parser';

const session = createSession({
  allowHttp: false,
  maxDepth: 32
});

const result = await session.parseDocument('/path/to/tokens.json');

if (result.diagnostics.hasErrors()) {
  // Format diagnostics for display or CI output
}

const resolution = result.resolver?.resolve('#/color/brand/primary');
if (resolution?.token) {
  console.log(resolution.token.value);
}

Parse sessions expose:

• document: the decoded RawDocument including original text and source map.
• ast: the normalised AST when schema validation succeeds.
• graph: the constructed document graph for pointer lookups.
• resolver: a DocumentResolver that evaluates aliases, overrides, and fallbacks.
• extensions: plugin evaluation results.
• diagnostics: a DiagnosticBag populated by all pipeline stages.

Token helpers

Use parseTokens when you want flattened tokens, metadata, resolution traces, and normalised diagnostics in one call:

import { parseTokens } from '@lapidist/dtif-parser';

const result = await parseTokens('tokens/base.tokens.json', {
  onDiagnostic: (diagnostic) => {
    if (diagnostic.severity === 'error') {
      console.error(diagnostic.message);
    }
  }
});

console.log(result.flattened.length);
console.log(result.metadataIndex.get('#/color/brand/primary')?.description);
console.log(result.resolutionIndex.get('#/color/brand/primary')?.value);

The synchronous variant consumes inline content without touching the filesystem. It throws if a loader would be required:

import { parseTokensSync } from '@lapidist/dtif-parser';

const { flattened } = parseTokensSync({
  $schema: 'https://dtif.lapidist.net/schema.json',
  values: {
    color: {
      brand: { primary: { $type: 'color', $value: '#0055ff' } }
    }
  }
});

console.log(flattened[0]);

CLI usage

The CLI wraps the same session pipeline and is useful for quick inspections:

npx dtif-parse tokens.yaml --resolve "#/color/brand/primary" --format json

Key flags:

• --resolve <pointer> – resolve one or more JSON pointers.
• --context key=value – provide override context values.
• --allow-http – enable HTTP(S) loading in the default loader.
• --max-depth <number> – cap the resolver depth (default 32).
• --format json – emit a machine readable summary instead of the pretty printer.

Token helper APIs

The token helpers sit on top of the session pipeline and add caching, flattening, and metadata snapshots. They support both asynchronous and synchronous workflows without forcing serialisation round trips.

Supported inputs

parseTokens accepts the same sources as parseDocument while adding explicit support for DTIF objects:

• string – interpreted as a URI or filesystem path. The default loader reads from disk and resolves relative paths.
• URL – parsed with the configured document loader (HTTP, file, or custom).
• { contents: string; uri?: string } – inline text with an optional URI for diagnostics.
• ArrayBuffer, Uint8Array, or Buffer – treated as raw bytes paired with the provided URI, if any.
• DesignTokenInterchangeFormat object – consumed without serialisation. The loader builds a synthetic handle so schema validation and caching reuse it.

The synchronous helper only accepts inline text or DTIF objects. It raises an error if resolving the input would require I/O (such as HTTP or disk access).

Options

ParseTokensOptions extend the base session options with flattening, graph control, caching hooks, and diagnostic observers:

• flatten (boolean, default true) – skip the flattening stage when false, returning empty arrays for flattened, metadataIndex, and resolutionIndex.
• includeGraphs (boolean, default true) – include document, graph, and resolver on the result. Disable when you only care about the flattened outputs.
• cache (ParseCache) – stores flattened tokens, metadata, resolution snapshots, and diagnostics keyed by the document hash. The bundled InMemoryParseCache offers an LRU eviction policy.
• documentCache (DocumentCache) – shares decoded documents across sessions. Async-only: parseTokensSync throws if this is provided.
• onDiagnostic ((diagnostic: TokenDiagnostic) => void) – invoked for every diagnostic in severity order as soon as it is produced, including cache hits.
• warn ((diagnostic: TokenDiagnostic) => void) – called for non-fatal warnings. Use this to surface soft failures immediately while still receiving a complete result.
• ...ParseSessionOptions – any session option (loader, schemaGuard, plugins, etc.) is forwarded to createSession.

Result shape

Both helpers return the same structure:

interface ParseTokensResult {
  document?: RawDocument;
  graph?: DocumentGraph;
  resolver?: DocumentResolver;
  flattened: readonly DtifFlattenedToken[];
  metadataIndex: ReadonlyMap<TokenId, TokenMetadataSnapshot>;
  resolutionIndex: ReadonlyMap<TokenId, ResolvedTokenView>;
  diagnostics: readonly TokenDiagnostic[];
}

• document, graph, and resolver are only present when includeGraphs is true and the document parsed successfully.
• flattened provides ready-to-render values, aliases, and references.
• metadataIndex exposes per-token metadata for descriptions, extensions, and deprecation details.
• resolutionIndex mirrors the resolver cache so you can inspect resolution paths, applied overrides, and reference chains.
• diagnostics is always populated, even for cache hits. The array is ordered by severity, then by the original emission order.

Flattened token entries

DtifFlattenedToken aligns with the DTIF schema. Each entry includes:

• id: Stable token identifier, matching the canonical JSON pointer.
• pointer: Pointer to the token’s location within the source document.
• type: The declared DTIF token type.
• value: Resolved JavaScript value suitable for design tooling or build pipelines.
• raw: The un-transformed $value from the source document.
• path: Collection path segments that describe the hierarchy.
• mode: Normalised mode value when present.

Metadata snapshots

createMetadataSnapshot(session, graph) builds the metadataIndex map and is exported for advanced use cases. Metadata entries are cloned to plain JSON and contain:

• description: Normalised description text when present.
• extensions: Deep-cloned extension records.
• deprecated: Optional object describing deprecation reasons and replacement pointers.
• source: { uri, line, column } pointing to the token definition for diagnostics and editor integrations.

Resolution views

createResolutionSnapshot(resolver) powers resolutionIndex. Each entry tracks:

• id, type, and raw schema-level information.
• value: The resolved token value, post-aliasing and overrides.
• references: Immediate references encountered while resolving the token.
• resolutionPath: The ordered chain of pointers the resolver followed.
• appliedAliases: Alias tokens that were ultimately applied.

Diagnostics

The parser never throws for document issues. Instead, each stage records diagnostics in a DiagnosticBag. Use result.diagnostics.toArray() for raw access or the helper methods hasErrors(), count(), and filter() to categorise messages. CLI output mirrors this information.

All diagnostics emitted by the loader, schema guard, normaliser, and resolver are normalised via toTokenDiagnostic. The TokenDiagnostic interface is compatible with Language Server Protocol conventions and includes related information with URI-anchored ranges. Format diagnostics for terminal output or logging with formatTokenDiagnostic:

import { parseTokens, formatTokenDiagnostic } from '@lapidist/dtif-parser';

const { diagnostics } = await parseTokens('tokens/base.tokens.json');

for (const diagnostic of diagnostics) {
  console.log(formatTokenDiagnostic(diagnostic, { color: process.stdout.isTTY }));
}

parseTokens invokes onDiagnostic and warn hooks immediately so tooling can stream feedback while the document is processed. Cached results re-emit the saved warnings before returning.

Working with caches

The default behaviour performs no caching. Provide a cache to speed up repeated parses:

import { createSession, InMemoryDocumentCache } from '@lapidist/dtif-parser';

const cache = new InMemoryDocumentCache({ maxAgeMs: 60_000, maxEntries: 100 });
const session = createSession({ cache });

Caches receive decoded documents and are responsible for TTL (maxAgeMs) and eviction policies. The parser validates cached bytes before reuse to avoid stale results.

The token helpers separate document caching from flattened artefact caching:

import { parseTokens, InMemoryParseCache } from '@lapidist/dtif-parser';

const cache = new InMemoryParseCache({ maxEntries: 50 });
await parseTokens('tokens/base.tokens.json', { cache }); // parses and caches
await parseTokens('tokens/base.tokens.json', { cache }); // served from cache

createParseCache exports the same implementation with configuration defaults. Provide your own cache by implementing get and set if you need persistence or shared storage.

Loader configuration

DefaultDocumentLoader resolves inline content, filesystem paths, and optionally HTTP(S) URLs. It enforces a 5 MiB default byte cap. Override the loader to integrate custom protocols:

import { createSession, DefaultDocumentLoader } from '@lapidist/dtif-parser';

const defaultLoader = new DefaultDocumentLoader();

const session = createSession({
  loader: {
    async load(input, context) {
      if (typeof input === 'string' && input.startsWith('memory://')) {
        return {
          uri: new URL(input),
          bytes: fetchFromMemory(input),
          contentType: 'application/json'
        };
      }
      return defaultLoader.load(input, context);
    }
  }
});

Plugins

Plugins extend the parser with $extensions collectors and resolution transforms.

import { createSession } from '@lapidist/dtif-parser';

const session = createSession({
  plugins: [
    {
      name: 'example.extensions',
      extensions: {
        'example.extensions'({ value }) {
          // validate extension payload
          return { normalized: value };
        }
      }
    },
    {
      name: 'example.transforms',
      transformResolvedToken(token) {
        if (token.type === 'color') {
          return { data: { rgb: convertToRgb(token.value) } };
        }
      }
    }
  ]
});

Use createPluginRegistry when you need to reuse a plugin set across sessions or feed the normalised transforms into manual resolver construction.

Extension collectors run during normalisation; transform plugins run after resolution. Both may add diagnostics that flow into the session result.

Integration tips

• Reuse sessions across documents to take advantage of shared caches and schema guards.
• Enforce sensible maxDepth limits when resolving user-provided documents.
• Persist the serialised parse result (document, ast, and graph) if downstream tooling relies on consistent pointer structures.
• Normalisation emits plugin evaluations and metadata so IDEs and design tools can surface extension information without re-parsing.

Node adapter

The Node adapter wraps the helper in filesystem-friendly ergonomics:

import { parseTokensFromFile } from '@lapidist/dtif-parser/adapters/node';

try {
  const result = await parseTokensFromFile('tokens/base.tokens.json', {
    cwd: process.cwd(),
    onDiagnostic: (d) => console.log(d.message)
  });
  console.log(result.flattened.length);
} catch (error) {
  if (error instanceof DtifTokenParseError) {
    for (const diagnostic of error.diagnostics) {
      console.error(formatTokenDiagnostic(diagnostic));
    }
  }
}

• Validates file extensions against the .tokens.json convention.
• Normalises diagnostics to TokenDiagnostic values and exposes them on DtifTokenParseError.
• Populates the same flattened token and snapshot structures returned by parseTokens.
• Provides readTokensFile when you only need the decoded DTIF document.

Package architecture

Understanding how the codebase fits together makes it easier to extend the API without breaking existing workflows.

Public entry points

The package exposes all user-facing APIs from the main index module:

• src/index.ts re-exports factories and utilities including createSession, parseDocument, resolver helpers, graph traversal utilities, diagnostic primitives, and the token helpers documented above.
• src/session.ts defines the ParseSession class and the createSession factory. Sessions orchestrate document loading, caching, schema validation, AST normalisation, graph construction, and resolver initialisation for each parse request.

Session lifecycle

ParseSession.parseDocument executes the pipeline below:

1. Use the configured document loader to convert caller input into a DocumentHandle, surfacing loader diagnostics on failure.
2. Resolve cached RawDocument instances before decoding handles via decodeDocument, again collecting diagnostics when decoding fails.
3. Validate the JSON schema through SchemaGuard, returning early when the document is invalid while still exposing decoded bytes to the caller.
4. Normalise the AST and build the directed token graph, feeding diagnostics back into the shared DiagnosticBag.
5. Instantiate a DocumentResolver that supports alias resolution, overrides, and plugin transforms whenever a graph is available.

The parseTokens helper reuses these seams so it can share infrastructure with existing session workflows while layering flattened outputs and metadata snapshots on top.

Graph and resolver utilities

Supporting modules wire the AST into usable graph and resolver structures:

• src/graph/builder.ts generates a DocumentGraph from the normalised AST and emits structural diagnostics when relationships cannot be resolved.
• src/resolver/document-resolver.ts implements the resolution engine. It tracks visited pointers, enforces the maxDepth guard, records override applications, and exposes ResolvedToken instances for downstream tooling. The resolver already caches intermediate resolution state that snapshot helpers reuse.

Diagnostics and supporting types

Diagnostic primitives keep feedback consistent across the loader, normaliser, and resolver:

• src/diagnostics/bag.ts collects parser, loader, and resolver diagnostics in a stable insertion order while offering convenience helpers such as hasErrors and severity counters.
• src/diagnostics/codes.ts and src/diagnostics/severity.ts define the severity taxonomy and stable diagnostic codes that map onto the TokenDiagnostic shape.
• src/types.ts centralises shared type definitions including ParseInput, ParseResult, DocumentGraph, and the diagnostic model.

Testing conventions

Tests live alongside the source under parser/tests and are split by scope:

• Unit suites in tests/unit exercise isolated components such as session caching, resolver alias chains, snapshot builders, and diagnostic helpers.
• Integration suites in tests/integration cover multi-file parsing, end-to-end parseTokens usage—including cache reuse and synchronous inputs—plus the Node adapter surface.

Keeping this reference close to the rest of the documentation ensures ongoing roadmap work builds on a shared understanding of both the public API and the parser internals.

Related resources

• @lapidist/dtif-schema
• @lapidist/dtif-validator
• dtif repository