Lance v14 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a file format, a table format, index formats, catalog specs, and a namespace client spec. The Rust workspace at lance-format/lance implements all of them plus Python (pylance) and Java bindings.

This skill tracks v14.0.0-beta.14 (the lance-format/lance git tag), the current development frontier; v13.0.0 is the stable pin, released 2026-10-07. Pin against tags, not main - Lance ships beta tags every few days and next-format encodings can change. Version landscape below.

Three layers of reference, load what the task needs:

• The deep reference - any concrete schema, parameter, proto, or constraint. Split by topic:

  File in references/	Covers	Sections
  format-file.md	What Lance is, the 27 crates, file format, data types	1-4
  format-table.md	Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL	5-10
  indexes.md	Vector / scalar / FTS / geo indexes, distributed builds	11-12
  ops.md	Object store, capability matrix, source map	13, 15, 16
  changelog-v7-v14.md	The full v7 -> v14 delta	14

  Cross-references written as "section N" resolve through references/lance-reference.md.

• references/performance.md - ALL performance guidance. Part A routes to the official text and adds the source-derived changes upstream has not documented; Part B is field-verified remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.

• references/docs/ - a verbatim mirror of the official docs (docs/src at the tracked tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Directory map below.

references/maintenance.md covers refreshing this skill against a new upstream tag.

Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

• Lance - the format and engine. The lance-format/lance repo; the lance /lance-* Rust crates; pylance. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. This skill is about Lance.
• LanceDB - a separate database product (lancedb/lancedb) built on top of Lance. It adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here.

The wider ecosystem (separate repos, own version lines, none covered here): Flink streaming writes (lance-flink), PostgreSQL reads via pglance, a Cypher graph engine (lance-graph), a dataset browser (lance-data-viewer), agentic context management (lance-context), and namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue. The canonical docs site is lance.org. Generated per-language SDK docs live at lance-format.github.io/lance-python-doc for Python and javadoc.io for Java - the matching lance-format.github.io/lance-java-doc path 404s.

Linking the lance crate in Cargo.toml means you are using Lance directly - use this skill. For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the authority for the format itself.

The wrapper can hide format features. LanceDB's create_table cannot enable stable row IDs; only pylance's write_dataset(enable_stable_row_ids=True) can. If a format-level capability matters to your design, check whether the wrapper exposes it before assuming the underlying format settles the question - and reach for pylance directly when it does not.

The crate workspace

27 crate directories under rust/ at v14.0.0-beta.14 - #9534 vendored the jsonb codec as lance-jsonb (26 at v13.0.0). lance is the public entry point - Dataset, scanner, indexes, commits; everything else (lance-table, lance-file, lance-encoding, lance-index, lance-io, lance-core, lance-datafusion, lance-linalg, lance-namespace*, ...) is a layer beneath it. Edition 2024, arrow 58, datafusion 54, pyo3 0.28; Python bindings need 3.10+. MSRV is 1.94.0 in the v14 line (#8997), up from 1.91.0 through v13.0.0. Full table with roles, versions, and every workspace dep in references/format-file.md section 2.

If you depend on anything below lance, expect moves. v11 deleted lance-encoding::version with no re-export (LanceFileVersion and ConcreteFileVersion live in lance-file::version), removed lance_io::encodings, and gave each format its own versions/v2_{0,1,2,3} module (#8020-#8026). The transaction code moved to a rust/lance-table/src/transaction/ module tree (#8053/#8054/#8056); a lance::dataset::transaction shim still re-exports Operation, Transaction, TransactionBuilder, RewriteGroup and UpdateMap, so the common surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs retargeting. Section 2.1.

File format versions

The file format carries a single major.minor version. data_storage_version is set per dataset at creation - but since v12.0.0 it is no longer fixed once the dataset exists. It is the default for writes that omit a target: "Create and overwrite establish this default; append, update, merge-insert, and compaction do not change it." An existing V2 dataset can take "2.0", "2.1", "2.2" or "2.3" per operation without rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2 versions. V1 and V2 still cannot be mixed. Section 3.

stable resolves to 2.2 (#8657), and 2.2 is the enum #[default], so a dataset created without an explicit data_storage_version is written as 2.2 - Python's write_dataset inherits it. The docs still do not say so - format/file/versioning.md only calls stable an "alias for the default version" - so the code is the authority. next resolves to 2.3. Pin an explicit number for deterministic behavior across builds: anything relying on WriteParams::default() (tests, benches) silently changed format at the 2.2 switch.

On v12, a writer defaulting to 2.2 rejected legacy blob metadata outright. Since v13.0.0 (#9459) legacy blob metadata written to 2.2+ is converted to Blob v2, and a 2.0/2.1 table can append Blob v2 without a rewrite. Blob v2 tiers by size: 64 KiB inline, 4 MiB dedicated, packed .blob sidecars between (the pre-v14 docs said 16 KiB / 2 MiB; the code never did).

The release selectors (LanceFileVersion) are a type distinct from the persisted identity (ConcreteFileVersion). Details in references/format-file.md sections 3.1 and 3.6.

Version landscape

The major is bumped by a bot, not a human: ci/publish_beta.sh re-roots at MAJOR+1 whenever any PR since the release root carries the GitHub breaking-change label - the marker is the label, not a conventional-commit !. A major bump therefore means "some labeled breaking change landed", not a redesign. It has fired on five consecutive lines: none of v9.1.0, v10.1.0, v11.1.0, v12.1.0 or v13.1.0 was released. 13.1 got further than the rest - v13.1.0-beta.1 and beta.2 were tagged (fury 13.1.0b1/b2) before #9518 re-rooted it to 14

• so a 13.1 beta pin is a dead end, not an early 13.1.

Four recent lines did ship a final: v10.0.0 (2026-08-08), v11.0.0 (2026-08-30), v12.0.0 (2026-09-17) and v13.0.0 (2026-10-07). Each sits on a stabilization branch that is not an ancestor of main - normal for a Lance final. v13.0.0 is the 14 release root plus two release commits, so every v13-final change is also in every v14 beta.

v13.0.0 is the stable pin and what GitHub Releases marks Latest. crates.io carries finals only (newest lance 13.0.0); PyPI pylance is likewise at 13.0.0. A beta pin means a git dependency - beta wheels publish to fury.io (https://pypi.fury.io/lance-format), which carries pylance-14.0.0b1 through b14.

Full per-tag deltas with every PR citation: references/changelog-v7-v14.md.

Manifest feature flags

A reader or writer refuses a manifest that sets a bit it does not support. The supported set is FLAG_UNKNOWN - 1 minus whatever the build masks out, so the docs table is not the authority - rust/lance-table/src/feature_flags.rs is. Bits 0-6 are long-standing (deletion files, stable row IDs, the deprecated v2 marker, table config, base paths, disabled transaction file, and the env-gated data overlay files). Everything from bit 7 up arrived in the last four majors, and FLAG_UNKNOWN moved 1 << 9 (v12.0.0) -> 1 << 14 (v13.0.0) -> 1 << 16 (v14 line):

"Sticky" means the bit survives every later commit (STICKY_PAIRED_FLAGS / STICKY_WRITER_FLAGS), so setting it once permanently raises the reader floor of the table.

Two pin windows open a table they should refuse - or brick it for older readers.

• Do not pin v11.0.0-beta.4 through beta.17. Those builds read bit 7 as a retired MemWAL flag they support, so they open a covering-index dataset instead of refusing it - wrong neighbours, no error.
• Do not pin v14.0.0-beta.4 through beta.13. Any ordinary Blob v2 write there sets FLAG_MANAGED_BLOBS, which is sticky, so v13.0.0 can never open the table again. #9891 (beta.14) made it explicit opt-in - "Ordinary Blob V2 writes currently activate FLAG_MANAGED_BLOBS and require newer readers" - but the beta.14 docs still describe automatic activation. Trust the code.

Covering indexes (#8535) - IndexMetadata.covering_fields (proto field 11) names the columns an index carries values for, so a query projecting only those is answered without a base-table take. Under the contract every released build implements, the carried columns are the trailing suffix of fields in emission order, and a column the index is both keyed on and carries appears twice: "fields = [1, 1] with covering_fields = [1] is keyed on 1 and carries 1." #9159 specified an independent contract behind bit 13 (fields holds only keys), but the build masks that bit, so it is spec only. Invalidation is wide either way: any index whose fields include the updated column. V3 IVF auxiliary files can physically carry columns ("a reader discovers carried columns by exclusion, not by position"), bound by a covering_field_ids metadata key, and coverage is per-segment. Section 11.

The v13 final and v14 delta

161 commits from v13.0.0-beta.4 to the v13.0.0 final, then 137 more on the v14 line to beta.14. 8 breaking-change-labeled PRs: #7905, #9479, #9551, #9547, #9159 in v13 final; #9518, #9534, #8331 in v14. Invariants: 27 crates (v14), 17 transaction ops (Operation::Unknown, v14), CommitConfig.num_retries still 20 (in v14 the retry clock starts after catch-up, #9631), arrow 58 / datafusion 54.

The label missed the format break again. #9482 (v14.0.0-beta.2) moved the inline transaction from manifest field 21 to field 23, added Operation::Unknown so unknown operations decode instead of failing, and turned From conversions into TryFrom - labeled format-change, never breaking-change, so the bot did not count it. Exhaustive match on Operation fails to compile.

• merge_columns field ids come from the manifest (#9547) - the Rust max_field_id argument is gone. The bug it fixed made merged columns silently read back as null.
• BM25F cross-field search (#7905) - combined_fields / CombinedFieldsQuery scores one query across several FTS-indexed columns as a single field. Section 11.
• JSON API moved (#9479, #9534) - one accessor (JsonTextStream::try_new), jsonb_to_json removed, and jsonb vendored as lance-jsonb with its own spec (format/table/jsonb.md). The v13 rule still stands: only json_get_int / _float / _bool / _string reach a JSON index (#9101); json_extract / json_get silently full-scan.
• ScalarIndexPlugin::load_index takes index_version (#9518) - every third-party scalar index plugin fails to compile.
• Duplicate-pair enumeration (#9493) changed output order and cosine values in #9551.
• Default IVF probing moved, unlabeled. #9184 leaves probes unset so queries get adaptive bounds, #9467 recalibrated dot-product profiles, and #9798 (beta.12) added search_effort (0 to 1, resolved to 0.5; it overrides a fixed nprobes) and dropped the Python binding's DEFAULT_NPROBES = 1 minimum-probes floor. Latency and recall at default settings shift across these upgrades - re-benchmark rather than carry numbers forward.

Net-new format surface: tagged FRI is supported (bit 10, #9420), extending reuse to reclustering via StablePartition; the fragment metadata tree spec (#9060 - fragment_metadata.proto, Manifest field 22, a _bt/ directory; bit 12 still rejected); spilled row lineage replacing the retired DataFragment.external_* fields (#9253, hidden columns at field ids -3/-4/-5, unstable); non-reusable field ids (#8658, bit 15, max_allocated_field_id); and an experimental MinHash LSH index (type 12, #9076/#9114).

Behavior changes without a label: explicit cloud storage options now beat env vars (#9719); fast_search no longer adds _rowid (#9699); main is reserved as the default-branch alias (#7180); Restore conflicts with an UpdateConfig touching schema or field metadata (#9361); raw deep-Clone commits are rejected (#9555); the DataFusion memory pool is 150MB per effective partition - all cores when target_partitions is unset - with per-execution pools (#9183, #9408).

Fixes needing a rebuild, rewrite or repair, not just an upgrade: rewrite data hit by #9018, #9239 (NULL written as 0) or #9394; rebuild scalar indexes hit by #9421 (segment merge produced an empty index), #9352 (LABEL_LIST) or #9203; repair or restore after #9218, #9219, #9573, #9447, #9176, #9595, #9085; v14: #9760, #9524, #9620, #7706, and #9857 (MemWAL, beta.9-12 only). Address-domain indexes - ZoneMap, BloomFilter and FM-Index - still lose coverage of compacted fragments rather than claim them (v14 recalculates instead when a fragment reuse index is present, #9514), and one damaged under v10 or earlier needs recreating. Conditions for each in references/changelog-v7-v14.md.

Still not landed: the arrow 59 / DataFusion 55 / pyo3 0.29 upgrade. #8997 merged only as "prepare" ("Not in this PR: the dependency bumps and lockfiles"); the real bump is #9647, open and blocked on upstream releases. It also gates PyO3 RUSTSEC-2026-0176/0177, still outstanding.

Earlier majors in brief

v13 betas - WriteParams.file_writer_options (#9192, struct literals fail to compile); lazy page-metadata init (#7465): StructuralFieldScheduler::initialize takes requested_ranges, and caching moved from per-column FieldDataCacheKey to per-page PageDataCacheKey, so every warm or persisted metadata cache cold-misses across the upgrade - the payoff is that "a cold point/range read's metadata IO is invariant to the column's total page count". The FRI got a versioned on-disk contract (#9136; stable row IDs and tagged FRI mutually exclusive outside debug builds). inline_optimization_enabled flipped true -> false (#9180). hf:// object store (#9236).

v12 - mixed data-file versions shipped (#8581-#8585); stable -> 2.2 (#8657); IVF_RQ defaults to 5 bits, ~4.4x the 1-bit index size (#8936; set num_bits=1 to opt out); WrappingObjectStore implementors must add wrap_paginated - "There is deliberately no default"

• and paged listing via read_dir_page means walking until the token is None, never until a page comes back short; namespace merge-insert keys have arity-dependent NULL semantics (a composite key's NULL "matches nothing"); lance-namespace pins now differ by language (Rust 0.13.0, Java 0.12.0 since #8979, Python >=0.13.0,<0.14 in v14 - re-check rather than carry forward).

v11 - fragment ids are a dataset-lifetime high-water mark (#8206): overwrite no longer restarts at 0, and datasets with duplicate ids read but no longer commit. LanceFileVersion lost Ord (#8027/#8028). Operation::Project / Merge gained preserves_nullability (#8347). The external-manifest protocol made object storage authoritative (#8499) - a retained ETag makes readers reject a good manifest with Manifest e_tag mismatch. #8101 silently duplicated rows on nullable primary keys under repeat merge_insert; existing duplicates must be removed by hand. Rebuild HNSW indexes hit by #8834 (v11) or #9053 (v13 betas).

Performance questions

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index maintenance cost, memory sizing, version bloat, benchmarking - load references/performance.md first. Part A routes to the official guidance plus the undocumented source-derived changes; Part B is field-verified practice against S3-compatible storage. The governing rule stays minimize remote calls - fewer commits, fewer scans, fewer round trips - because that is where the order-of-magnitude wins are. The official "Tuning remote scans" section gives a starting point for cross-region or public-internet access, where the cloud default of 64 concurrent requests is too aggressive: LANCE_IO_THREADS=8, fragment_readahead=1, batch_readahead=2, io_buffer_size=64MB. It is a legitimate second move once call volume is already minimized.

AMX-FP16 (#8540) is the one performance change that alters results, not just speed: where it engages, IVF partition assignment becomes exact instead of approximate, so recall improves and assignments differ from an older build. It is shape-gated (float16 + dot, dimension >= 32, num_centroids >= 32). LANCE_DISABLE_AMX=1 disables it and reverts assignment to the approximate path too.

Two cache facts to know before tuning anything remote: Lance has no resident data cache (a Session holds only index and metadata caches, never decoded values, so repeated point reads re-pay object-store IO), and one Arc<Session> shared via DatasetBuilder::with_session lets datasets share it. Cold first search is dominated by paging indexes in - prewarm_index is the remedy, and for FTS it loads the whole inverted index into RAM. #7465 changed the metadata cache key shape, so the first run after a v13 upgrade re-pays that paging even against a warm cache.

Layout debt is version-dependent. Binary-copy compaction of small appends preserves tiny pages; a reader that pays per page (v12) turns that into tens of thousands of GETs per lookup. Re-encode append-heavy tables, and benchmark on a fresh copy of the real store - details and measurements in references/performance.md Part B.

Time travel is not an archive mechanism. Versions look like free history, but the default cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design that treats old versions as the durable record loses it on the first maintenance pass. Keep an explicit archive if you need one. The reverse also holds: delete alone does not erase data - old versions, unrewritten fragments, and indexes (deletes do not conflict with index creation) all keep it until rewritten, cleaned up, and rebuilt.

Official docs mirror

references/docs/ mirrors docs/src of lance-format/lance at the tracked tag, verbatim - 49 markdown files plus 5 diagrams, all directly readable.

Not mirrored: docs/src/images/ and the format/*.png assets, so image links in the mirrored pages do not resolve - the prose is self-contained, and the five .svg diagrams are mirrored. Also out by design: community/, examples/, integrations/{index,pytorch,tensorflow}.md; and the landing stubs and contributor files (format/AGENTS.md).

A whole tier of docs is not in this repo at all, so it cannot be mirrored and cannot be enumerated from a clone. docs/make-full-website.sh assembles format/catalog, format/namespace, and the integrations/{duckdb,huggingface,spark,ray,trino,context} sections at build time from six sibling repos with their own version lines - the checked-in integrations/index.md links spark/, duckdb and trino as if they were local, but those paths do not exist in the tree. Lance Context and the HuggingFace integration docs are whole nav sections that exist only on the built site. For any of those, read lance.org rather than this mirror. Protobuf message bodies are likewise expanded at build time from protos/ by mkdocs_protobuf, so the mirrored spec pages show %%% proto.message.X %%% placeholders where the site shows a rendered schema.