Brotli import and export

export({history:'full'}) and its streaming equivalent write manifest 3, which records source lineage, history coverage, original DBSONL commit receipts and schema events. A full restore validates and rebuilds historical projections atomically in an empty target. Default exports write manifest 2 with current documents and ordinary index definitions. Full backup and restore

Database.export(), Database.exportStream() and Database.import() use defuss-brotli 0.3.0 and binary defuss-dson. Explicit snapshots also expose both export methods. The portable archive contains collection schemas and documents; equality, ordered and lexical/vector/hybrid indexes are rebuilt on import. Native values, cycles and shared references within a document retain DSON semantics. No embedding model or custom analyzer function is serialized.

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { openDirectory } from 'defuss-db/server';

const source = await openDirectory('./source');
await pipeline(source.exportStream({ quality: 4 }), createWriteStream('backup.dfdb.br'));
await source.close();

const target = await openDirectory('./restored');
const receipt = await target.import(createReadStream('backup.dfdb.br'));
console.log(receipt.imported, receipt.sourceRevision, receipt.revision);
await target.verify();
await target.close();

Browser workers use the same archive API on openOPFS() handles:

const bytes = await db.export({ collections: ['documents'], quality: 4 });
// Store/download bytes as application/octet-stream, or pass a Blob/File stream directly.
await restored.import(new Blob([bytes]).stream());

The byte-array method copies import input synchronously before its first await. Stream chunks are copied when pulled, so a producer may reuse a buffer after its next pull. Accepted import inputs: Uint8Array, synchronous/asynchronous iterables of byte arrays, and ReadableStream<Uint8Array>.

Consistency and conflicts

Limits and memory

Option Default Scope
quality 4 Export; integer 0–11
blockBytes 1 MiB Export target block size; minimum 1 KiB
maxFrameBytes 32 MiB Both compressed and decoded frame sizes; configurable 1 KiB–256 MiB
maxArchiveBytes 1 GiB Streaming export/import total compressed bytes
buffered export().maxArchiveBytes 64 MiB Buffered export total compressed bytes
maxDecodedBytes 1 GiB Import total uncompressed frame bytes
maxDocuments 1,000,000 Both directions
maxCollections 10,000 Both directions

All methods accept signal. Export additionally accepts a unique list of existing collection names. Import also honors the target's DBSONL record, transaction-byte and transaction-operation limits, including derived index writes. Raise those limits explicitly for larger atomic imports.

Streaming avoids assembling the whole archive in memory. It does not make transaction staging constant-space: import retains staged writes until commit. Export holds bounded encoded blocks plus a bounded native scan batch; object heap size can exceed encoded size. Import also holds the producer's current chunk. Buffered export retains all compressed chunks and allocates the final joined array. A single large document may exceed blockBytes but must fit maxFrameBytes, including its archive envelope. Very large datasets should use appropriately sized budgets or application-level collection archives; no partial-commit import mode is implied.

Physical framing, version 1

The file starts with eight bytes 44 46 44 42 42 52 01 00 (DFDBBR, version 1, reserved zero). Each frame has a 44-byte header:

Offset Field
0 sequence number, unsigned 32-bit little-endian, beginning at zero
4 uncompressed payload length, unsigned 32-bit little-endian
8 compressed payload length, unsigned 32-bit little-endian
12 32-byte SHA-256 digest

The digest covers previousDigest || first12HeaderBytes || compressedPayload, starting with a zero-filled previous digest. Payloads are independent Brotli streams. Each decoded payload contains length-prefixed binary DSON records (u32le length, followed by the record bytes); records do not cross frame boundaries.

Latest-only archives use logical manifest 2 and a full-history archive uses manifest 3; import rejects any other manifest. Export copies logical documents and index definitions. Import preserves the target store UUID, creates or updates collection revision markers transactionally, and rebuilds new collections' equality, ordered and search indexes.

Records begin with a manifest, then sorted unique collection schemas, documents sorted by (collection, encodedPrimaryKey), and an end record with counts. A final zero-length frame with a valid chained digest is mandatory, followed immediately by EOF. Missing/reordered frames, trailing bytes and count mismatches are rejected. Length caps are checked before frame allocation/decompression, and the decoder's maxOutputSize is set to the declared bounded length.

Checksums detect accidental corruption. They are not authentication, signatures or encryption. The archive is a framed container, not one standalone .br stream; use db.import(), not a generic whole-file Brotli decoder.

Runtime and deployment

Compression/decompression modules initialize lazily. ESM and CommonJS consumers both load the actual package with dynamic import. The standard upstream distribution embeds its WASM, so no separate WASM asset URL is required. Browser bundling can still include that embedded payload; this release does not promise a small archive-enabled bundle.

Browser CSP must permit WASM compilation, e.g. script-src 'self' 'wasm-unsafe-eval'; JavaScript unsafe-eval is not required. OPFS still requires an appropriate secure worker context. Compression is synchronous within each block and can occupy its worker/event loop; use a dedicated worker for interactive applications. Multithreaded compression and background snapshot scheduling are not part of this API.

A full archive restores immutable history and source receipts into an empty target. Use a source-qualified revision to reproduce an earlier recorded answer after restore. Full backup and restore