Article
Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On
Cross-posted. This article's canonical home is iceberglakehouse.com.
Every roundup of Apache Iceberg v3 runs the same order: deletion vectors first, the Variant type second, and then, somewhere in the back half, a sentence about row lineage before the geospatial types close the show. The billing is understandable, deletes and Variant solve pains people already feel, and it is going to look wrong in retrospect, because row lineage is the v3 feature that changes what a table is. For the first time in the format's history, a row has an identity: a stable identifier assigned at birth, carried across every update, alongside a marker recording when the row was last touched. Identity is the primitive that change data capture, incremental computation, deduplication, auditing, and machine learning reproducibility have all been faking with application-level keys and timestamp guessing, and v3 builds it into the format itself, mandatory, for every table.
This article gives the buried feature the standalone treatment it deserves. We will establish why immutable-file tables never had row identity and what the absence cost, walk the two fields the specification adds and the genuinely clever inheritance mechanism that makes them nearly free to write, follow identity through updates, deletes, and compaction where the subtleties live, build the changelog recipe that turns two metadata columns into a CDC feed, survey what becomes possible downstream, and finish with the fine print, engine support unevenness, the identity-versus-key distinction, and the costs, because a feature this quietly foundational deserves precision rather than cheerleading.
Disclosure, as always: I work at Dremio, whose engine writes and maintains these fields automatically on v3 tables, and I co-authored O'Reilly's books on Apache Iceberg and Apache Polaris. Everything below is the open specification's story, quotable against the spec text.
Why the feature stays buried is worth a paragraph, because the burial pattern itself is informative. Deletion vectors and Variant improve workloads people run today, so their value lands on contact, while identity is infrastructure for workloads people are about to run, incremental everything, agentic access, changelog-driven pipelines, and infrastructure for the near future always demos worse than relief for the present. There is also nothing to configure, which starves the feature of the tutorial content that drives awareness: mandatory and automatic means no setup guide, no toggle to blog about, just two columns that appear when you look. Features like this get discovered the way plumbing gets discovered, when something downstream needs them, and this article exists so the discovery happens on your schedule rather than a deadline's.
The Problem: Rows Never Had Names#
Start from what was missing, because the absence was so normal nobody called it a problem.
An Iceberg table before v3 is a set of immutable files organized by snapshots, and a row, within that design, is a position: file such-and-such, offset so-and-so. Positions are coordinates, not identities. Rewrite the file, compaction, copy-on-write, a MERGE, and every row in it gets new coordinates while remaining, semantically, the same row, and nothing in the format records the correspondence. Ask the pre-v3 format "is this row in snapshot 50 the same row as that one in snapshot 40" and the honest answer is that the question is not expressible. The format tracks files and their changes exquisitely. Rows, the things people actually reason about, were anonymous.
Anonymity had a workaround, and the workaround defined a decade of pipeline architecture: identity by user-defined key. Pick columns that should uniquely identify a row, customer ID, order ID plus line number, and treat matching keys across snapshots as the same row. Every CDC diff, every deduplication job, every audit query ran on this convention, and the convention leaks everywhere keys do. Keys get reused, an order number recycled across years. Keys change, the migration that renumbered accounts. Keys are absent, event streams with no natural key. Keys are composite and expensive, five-column joins across billion-row snapshots to decide sameness. And keys are guesses about semantics that the storage layer cannot verify, so every downstream system built on them inherited the guess. As one of the feature's designers put it when v3 landed, before this, teams were reduced to guessing from user-defined identity columns, and now it is built into the format itself.
Count what the guessing cost in system design. Computing what changed between two snapshots, the foundational CDC operation, required either full-snapshot joins on the guessed keys, expensive and fragile, or a parallel infrastructure, sidecar changelogs, mirror tables, log-based capture bolted alongside the lakehouse, duplicating the data path to recover information the table technically witnessed and failed to record. Materialized view maintenance, which wants to recompute only from changed rows, fell back to coarser grains, changed partitions, changed files, or full refreshes. Idempotent reprocessing had no storage-level receipt for "already applied." The table format that gave the lakehouse transactions, time travel, and schema evolution left row-level change reconstruction to everyone's least favorite join.
The gap was also a maturity gap against the operational world, and practitioners crossing between the two felt it constantly. Databases have carried row identity forever, it is what a primary key plus internal row versioning amounts to, and every change-stream, trigger, and temporal-table capability rests on it, which is why the operational side generates precise change feeds so readily. The analytical side consumed those feeds and then, at its own layer, reverted to anonymity, able to receive changes and structurally unable to re-emit them with the same precision. Row lineage closes that asymmetry: the lakehouse table becomes a system that can testify about its own changes, at the grain the operational world always did, which is the precondition for the lakehouse serving as the middle of pipelines rather than only their end.
What Row Lineage Is: Two Fields and a Mandate#
The v3 answer is compact enough to state in spec language and then unpack for the rest of the article.
In v3 and later, an Iceberg table must track row lineage fields for all newly created rows. Engines must maintain a table-level field, the next row ID, and two row-level fields. The first is _row_id, a unique long identifier for every row within the table, assigned when the row is first added and stable thereafter: update the row ten times and its _row_id never changes. The second is _last_updated_sequence_number, the sequence number of the commit that last created or modified the row, so every row carries a when beside its who. The fields live in the reserved metadata column space, with reserved field IDs near the top of the ID range, and they surface to queries as ordinary selectable columns on engines that support them, no configuration, no opt-in, no schema declaration.
Three properties of the design deserve underlining before the mechanism, because each one was a decision.
It is mandatory. Row lineage began life as an optional v3 feature, switched on per table, largely because it initially looked incompatible with equality deletes, and once the coexistence question was resolved, the community moved deliberately to require it for v3, on the argument that standardized fields ease every developer's assumptions and the overhead is modest, two long columns that compress well. Mandatory is what makes the feature an ecosystem primitive rather than a per-table maybe: any v3 table, from any writer, carries identity, and downstream tooling can build on that without a capability negotiation per table.
It is identity, not a key. _row_id is a surrogate the format assigns, meaningless outside the table, unique within it, and deliberately unrelated to any business notion of sameness. The row for customer 42, deleted and re-inserted by a pipeline restart, is two identities with one key, and that distinction, which sounds pedantic, decides real semantics later in this article, so file it now: lineage tracks physical row continuity as the format performed it, and business identity remains the application's claim.
And it is retroactive never. Tables upgraded from v2 do not receive backfilled identities for pre-upgrade history: snapshots from before the upgrade have no first-row-ID bookkeeping, their rows read as null-lineage, and identity begins at the upgrade boundary. History starts when the historian was hired, which is the only honest option and worth knowing before you promise auditors otherwise.
A word on how the fields present themselves day to day, because their invisibility is deliberate and occasionally confusing. They are metadata columns: absent from DESCRIBE, excluded from SELECT star, never colliding with user columns thanks to the reserved ID space, and available the moment a query names them, the same contract as the file-path and position metadata columns practitioners already use for debugging. The presentation encodes the philosophy, identity is infrastructure, present everywhere, ambient, surfaced on request, and it has one practical consequence for tool builders: BI layers, quality frameworks, and export jobs that operate on "all columns" neither see nor ship the lineage fields unless explicitly told to, which is almost always the right default and worth knowing the one time it is not, the export that was supposed to carry identities downstream and silently did not.
The Inheritance Trick: How Identity Costs Almost Nothing to Write#
Here is the mechanism, and it is the part of the design that earns real admiration, because it threads a needle that looks unthreadable at first glance.
The needle: a writer producing data files cannot know the values these fields need. _last_updated_sequence_number should hold the commit's sequence number, and sequence numbers are assigned at commit time, by the catalog's atomic advance, after the files are already written, and a retried commit lands at a different sequence number than the attempt that lost the race. _row_id should hold globally unique values drawn from the table's next-row-ID counter, and the counter's position is likewise unknowable until the commit succeeds, since concurrent writers race for it. Writing correct values into immutable files before commit is impossible under optimistic concurrency, and rewriting files after commit defeats the entire architecture.
The spec's answer is inheritance: write null, and let null mean "compute me from context." A writer producing new data files leaves both fields null, physically absent, costing nothing but schema presence. At commit, the snapshot records a first-row-ID, set from the table's next-row-ID counter, and the counter advances by the number of rows the snapshot assigned, while each data file's manifest entry carries its own assigned first-row-ID within that allocation. A reader encountering a null _row_id computes it: the file's first-row-ID plus the row's position within the file. A reader encountering a null _last_updated_sequence_number inherits the file's data sequence number from the manifest. The values were never written because they never needed to be, they are derivable, deterministically, from metadata the commit machinery was maintaining anyway, and every reader derives identical answers because the derivation is specified.
Walk the allocation once to see the bookkeeping cohere. A table's next-row-ID stands at 600. A commit lands three new data files of 25, 25, and 50 rows. The snapshot takes first-row-ID 600, the files take assigned first-row-IDs 600, 625, and 650 through their manifest entries, rows within each file take consecutive IDs from their file's base by position, and the table's next-row-ID advances to 700 for whoever commits next. A concurrent writer that lost the race and retried simply inherits a later allocation on its successful attempt, correctness by construction, no coordination beyond the commit that was already atomic. Identity assignment rode the existing machinery and added, at write time, approximately nothing.
The design's admirable second half is the physical-versus-virtual flexibility. For fresh appends, the fields stay virtual, null on disk, derived on read, free. The moment derivation stops being possible, the fields materialize: an engine updating a row must carry its existing _row_id forward physically into the new file, because the new file's position arithmetic knows nothing of the row's origin, and a compaction rewriting files must write both fields as physical columns for every row it moves, preserving inherited values that position-based derivation can no longer reconstruct. Virtual when derivable, physical when not, and the boundary between the two is exactly the boundary between "row is where it was born" and "row has moved," which brings us to the interesting part of any identity system: what happens when things change.
Identity Under Change: Updates, Deletes, Compaction, and the Edges#
An identity system proves itself at the transitions, so walk each one, because the guarantees and their boundaries both live here.
An update preserves identity and refreshes the timestamp. When an engine updates a row, whatever the physical mechanism, copy-on-write rewriting the file, or merge-on-read marking the old position in a deletion vector and appending the replacement, the semantic contract is the same: the new version of the row carries the old _row_id, written physically into the new file, and takes a fresh _last_updated_sequence_number, the updating commit's. Ten updates later, one identity, ten timestamps in history, current timestamp visible. This is the contract that makes everything downstream work, and it is a contract engines implement, which is the honest phrasing: the specification defines how lineage is preserved, and an engine that processes an update as an unlinked delete-plus-insert produces a new identity where a continued one belonged. Mature v3 writers preserve, the reference implementation's paths preserve, and the engine-support section returns to the ones that do not yet, because a lineage chain is only as strong as the least careful writer on the table.
A delete ends an identity, and re-insertion does not resurrect it. The deleted row's _row_id simply stops appearing in subsequent snapshots, which is precisely how deletion becomes detectable, and a later insert of the same business key mints a fresh identity, because the format never knew about your keys. Pipelines with delete-then-reload habits, backfill jobs that clear and rewrite partitions, will see identity discontinuities across the reload boundary, correct by the format's lights and surprising to anyone who conflated _row_id with their key. The distinction filed earlier cashes out here: lineage answers "did the format carry this physical row forward," and only your keys answer "is this the same customer."
Compaction preserves identity by materializing it. A rewrite job consumes files whose rows carry inherited, virtual lineage, and produces files where derivation-by-position points at the wrong ancestry, so the spec obliges the compactor to write _row_id and _last_updated_sequence_number as physical columns in its outputs, carrying every row's inherited values across the move. Done right, compaction is invisible to lineage, the same identities, the same timestamps, new coordinates, and note what did not change: _last_updated_sequence_number stays put through compaction, because relocation is not modification, which is exactly the property that lets change-detection queries ignore maintenance churn instead of misreading every compaction as a mass update. Done wrong, a lineage-oblivious rewriter severs every chain it touches, and reading engines that require the bookkeeping fail loudly on files missing their first-row-ID metadata, a failure mode already visible in engine documentation and much preferable to silent identity loss. Add the shred-aware compaction requirement from the Variant story and the theme repeats: in v3, maintenance tooling is no longer generic Parquet plumbing, it is a spec-obligated participant, and "does your compactor preserve lineage" joins the vendor-evaluation checklist.
Deletion vectors and lineage compose cleanly, and the composition is worth stating because the features shipped together. A merge-on-read update marks the superseded position in the file's deletion vector and appends the replacement row, physically carrying the _row_id. The old file's masked row and the new file's row share an identity, exactly one of them visible per snapshot, and the vector's consolidation rule, one current vector per file, means reconstructing "which identities did this commit supersede" is a bounded comparison rather than an archaeology project. Row lineage plus deletion vectors is the pairing that makes v3's change story coherent: vectors make change cheap to record, lineage makes it precise to describe.
MERGE INTO, the workhorse of upsert pipelines, exercises every rule at once and is worth naming as its own case. Matched rows that update preserve their identities and refresh their sequence numbers, matched rows that delete end theirs, unmatched source rows insert with fresh allocations, and one MERGE commit produces, in lineage terms, exactly the mixed changelog its semantics describe, attributable row by row to a single sequence number. Concurrency composes too, through machinery this site has covered: identity allocation rides the commit, so racing writers cannot collide on IDs, a retried commit allocates at its successful attempt, and the concurrent-writer probe from the concurrency article gains a lineage clause, verify after the race that both writers' rows carry distinct, well-formed identities, which every conforming pairing delivers by construction.
And the equality-delete history explains a design scar worth knowing. Lineage was optional at first precisely because equality deletes, which remove rows by value without knowing which rows, sat awkwardly beside a system tracking rows individually, and the resolution of that incompatibility is what cleared the path to making lineage mandatory. The deeper alignment: the format's direction, lineage required in v3, equality deletes heading toward retirement in the v4 discussions, is one coherent position, that change should be attributable to identified rows, and mechanisms that cannot say which rows they changed fit the format's future poorly.
A Row's Biography, End to End#
Assemble the transition rules into one life story, because a single tracked row makes the whole system concrete. Follow order 8675309 through five commits on a v3 table whose next-row-ID stood at 4,000 when the story starts.
Commit at sequence 21, the insert. The order lands in a new data file at position 12. The writer wrote nulls for both lineage fields, the snapshot took first-row-ID 4,000, the file's manifest entry assigned it a base within that allocation, and readers derive the row's identity: _row_id 4,012, _last_updated_sequence_number 21, both virtual, costing zero bytes.
Commit at sequence 25, a status update, merge-on-read. The engine marks position 12 in the original file's deletion vector and appends the updated row to a fresh file, physically writing _row_id 4,012, preserved, and _last_updated_sequence_number 25, refreshed. The identity crossed files, the derivation stopped being possible, the materialization rule fired exactly on cue. A changelog between sequences 21 and 25 reports one UPDATE for identity 4,012, before and after status attached.
Commit at sequence 30, compaction. A rewrite job consolidates the region's small files, including both files from this row's history, masked original and live replacement, into one large file. The compactor, spec-obliged, writes the surviving row with physical _row_id 4,012 and _last_updated_sequence_number 25, unchanged, because relocation is not modification. A changelog spanning the compaction reports nothing for this row, which is the maintenance-invisibility guarantee observed in the wild.
Commit at sequence 34, a second update, copy-on-write this time. The row's file rewrites without it in the old form and with its new version carrying 4,012 and 34. Same identity, third timestamp, two physical mechanisms behind two updates, one semantic contract over both.
Commit at sequence 40, the delete. Identity 4,012 stops appearing. A changelog from 34 to current reports one DELETE, and when a reconciliation job re-inserts the order next week from the source system, the row returns as identity 4,713 or wherever the counter stands, a new biography for the same business key, which is the identity-versus-key line drawn in data rather than prose.
Five commits, one long integer, and every question the old world answered with key joins and guesswork, when did this change, what changed it, is this the same row, answered by two columns and the snapshot log. Pin the biography to the wall next to the changelog recipe, because between them they are the feature.
The Changelog Recipe#
Now the payoff mechanics: turning the two columns into a change feed with nothing but SQL over two snapshots. The pattern is the article's most practical asset, so here it is in full, comparing a table's state at an older snapshot against current.
WITH old_snap AS (
SELECT _row_id,
_last_updated_sequence_number AS seq,
*
FROM lake.sales.orders
VERSION AS OF 8271744332764321989 -- the older snapshot
),
new_snap AS (
SELECT _row_id,
_last_updated_sequence_number AS seq,
*
FROM lake.sales.orders -- current snapshot
)
SELECT
CASE
WHEN o._row_id IS NULL THEN 'INSERT'
WHEN n._row_id IS NULL THEN 'DELETE'
ELSE 'UPDATE'
END AS change_type,
coalesce(n._row_id, o._row_id) AS row_identity,
n.seq AS changed_in_sequence,
o.order_status AS before_status,
n.order_status AS after_status
FROM old_snap o
FULL OUTER JOIN new_snap n USING (_row_id)
WHERE o._row_id IS NULL -- inserts
OR n._row_id IS NULL -- deletes
OR n.seq > o.seq; -- updatesRead the query's logic against the semantics established above, because every line leans on a guarantee. An identity present only in the new snapshot is an insert, present only in the old is a delete, and present in both with a newer sequence number is an update, with the before-and-after values sitting on the same joined row, ready for downstream application. The join key is a single long, not a five-column business key, which changes the arithmetic of running this at scale, and the sequence-number filter is what makes the update detection exact rather than value-diffing every column. Compaction between the snapshots contributes nothing, same identities, same sequence numbers, filtered out by the update condition doing nothing, which is the maintenance-invisibility property earning its keep.
Refinements for production use follow the same grammar. Restrict the scan sides with partition predicates and the join shrinks to the slices that matter. Drive the "old" boundary from a stored watermark, the last sequence number a consumer processed, and the pattern becomes an incremental poll, each run picking up exactly the rows whose _last_updated_sequence_number exceeds the watermark, plus the disappearance check for deletes. And engines are increasingly packaging this exact pattern behind changelog functions and incremental read APIs, so the SQL above is both a tool you can use today and a description of what those conveniences do underneath, worth understanding even where a packaged version exists, because the packaged version's edge cases are this query's edge cases.
The recipe's limits are the semantics' limits, restated once as query behavior. Pre-upgrade history yields null identities, so changelogs begin at the v3 boundary. Intermediate states between the two snapshots collapse, a row updated five times between your endpoints appears once, final state, latest sequence, which is what most consumers want and not what a full event-history audit wants, that being a different pattern, walking snapshot pairs. And a writer that failed to preserve identity on update surfaces in this query as a spurious delete-insert pair, which makes the recipe, run against a known small change, a rather good conformance probe for your own writers.
The full-history pattern deserves its sketch, since audits ask for it. To reconstruct one row's every version, take the identity, take the snapshot log's sequence numbers within the audit window, and read the row at each snapshot where its _last_updated_sequence_number advanced, each advance being one version boundary, each read a time-travel query pinned by snapshot. The result is the row's biography as a result set, version, values, commit, and, joined through the log, timestamp and operation. It is a per-row query, priced for investigations rather than dashboards, and it converts "reconstruct what happened to this record" from a week of forensic joins into an afternoon of methodical SELECTs, which is roughly the difference between an audit finding and an audit estimate.
What Identity Enables, Domain by Domain#
With mechanism and recipe established, survey the payoff surface, because the title's claim, that AI and CDC workloads will come to depend on this, rests on the breadth here.
CDC sheds its sidecar. The pipelines that mirrored operational databases into the lakehouse always had the source half solved, log-based capture emits precise changes, and the sink half compromised, since applying changes into an identity-less table and then computing downstream changes back out of it meant key-join reconstruction or a parallel changelog store. With lineage, the table is its own changelog: apply the stream with v3's mechanics, vectors for supersession, lineage for continuity, and every downstream consumer reads changes with the recipe, no mirror infrastructure, no key guessing, the format finally holding the information it always witnessed. The pairing with deletion vectors is what makes high-churn CDC tables operable, and the pairing with lineage is what makes them consumable, and the two shipped together because they are two halves of one design.
Incremental computation gets its missing primitive. Materialized view maintenance, the aggregate that should update from changed rows rather than recompute from all rows, needs exactly what the watermark pattern provides: the delta since last processed, precisely scoped, cheaply retrieved. This is why platform engineering around incremental view maintenance names row-level tracking as a prerequisite, and why the ecosystem's convergence is telling, with Delta's row tracking serving the same role for its incremental machinery and every Iceberg v3 table carrying the capability by mandate. The materialized views, the derived tables, the feature pipelines that refresh on schedules today are the installed base for lineage-driven incremental refresh tomorrow, and the cost curve between "recompute the world" and "apply the delta" is the same proportionality argument this site has been making all year, now available at the row grain.
Idempotence gets a receipt. Exactly-once processing across restarts and retries has always needed somewhere to record "applied through here," and a watermark over _last_updated_sequence_number is that record, storage-level, engine-agnostic, surviving the consumer's own crashes. Reprocessing jobs, backfill reconciliation, cross-system sync all inherit a common grammar: identity says which row, sequence says which version, and together they make "have I seen this change" a lookup instead of a heuristic. Deduplication sharpens the same way: the classic dedup job groups on guessed keys and keeps an arbitrary winner, while a lineage-aware version distinguishes the genuinely duplicated ingestion, one business key, multiple identities, from the legitimately updated row, one identity, advancing sequence, and cleans the first without touching the second, a distinction the key-only version structurally cannot draw. Every pipeline that ever shipped a "dedupe carefully, updates look like duplicates" comment in its code knows exactly which distinction that is.
Audit and compliance get row-grain answers. When did this row last change, and in which commit, becomes a SELECT, joinable through the snapshot log to timestamps, operations, and, where platforms stamp them, committing principals. Walking a row's full history, its value at every snapshot where its sequence number advanced, becomes a bounded query over history rather than a forensic reconstruction. The caveats stay attached, history begins at the v3 boundary, lineage is physical continuity not business identity, but within those lines, the audit story moves from "we believe, based on our keys" to "the format records."
And the machine learning workloads, the title's second clause, get reproducibility and freshness at once. Training-set provenance, exactly which row versions produced this model, becomes recordable as a snapshot plus the lineage columns, and reproducible by time travel. Feature pipelines become incremental consumers, recomputing features for changed entities only, with the watermark pattern scoping each run. Embedding refresh, the wide-table pattern the column-family proposals target on the storage side, gets its bookkeeping side from lineage, which vectors' rows changed since the last index build being precisely a changelog question. And agent-driven data access, the audit-hungry newcomer, inherits row-grain attribution for free on every v3 table it touches: what the agent read has snapshots, what the agent changed has identities, and the compliance conversation about autonomous systems writing to governed data becomes tractable because the substrate records what happened at the grain the questions get asked.
The Fine Print#
A feature this useful earns a precise accounting of its edges, so here is the fine print, consolidated.
Engine support is the usual v3 patchwork, decomposed along the usual capability axes and one new one: reading the columns, preserving identity through updates and merges, materializing correctly through compaction, and, at the strict end, refusing to guess when bookkeeping is missing. The front of the pack writes and maintains the fields automatically across DML, the platforms that declared v3 general availability this year among them, engines are adding read support release by release, with the tracking issues public, and the trailing edge is the one to plan around: a writer that updates without preserving severs chains silently, which the changelog-as-probe trick catches, and a maintenance tool that rewrites without materializing breaks derivation, which strict readers convert into loud errors. The audit question for every tool touching a v3 table gained a fifth clause, and "reads, writes, deletes, planning, lineage" is the full 2026 litany.
The columns are omissible, by design, in one direction. The mandate is that v3 tables track lineage for new rows and that compliant writers maintain it, and the discussion that made lineage mandatory was explicit that the storage overhead is two well-compressing long columns, omittable as physical data wherever inheritance can derive them. The overhead conversation, in practice, rounds to a few percent on wide tables and to negligible after compression, cheap for what identity buys, and worth measuring on your narrowest, hottest tables where two longs are proportionally largest.
Identity is not ordering, and sequence numbers are not timestamps. _row_id values reflect allocation order loosely, concurrent commits interleave allocations, and nothing about the identifier is meaningful beyond uniqueness, so resist the temptation to sort by it or shard on it semantically. _last_updated_sequence_number orders commits, not wall-clock time, and maps to timestamps only through the snapshot log, a join worth building into audit tooling once rather than approximating everywhere.
Branches complicate the story exactly as much as you'd expect. Identity allocation rides commits, branches have their own commit lines, and rows created on a branch carry identities that merge machinery must reconcile with the main line's allocation, a corner the spec and implementations handle and your mental model should flag: changelogs computed across branch boundaries answer subtler questions than the same-branch recipe, and write-audit-publish workflows should compute their changelogs after the merge, on the line consumers read.
And the retroactivity boundary deserves its final restatement as a planning fact: lineage-dependent products, the CDC feed, the incremental refresh, the audit trail, date from each table's v3 upgrade, not from its creation, which argues for upgrading the tables whose future changelogs matter before those changelogs are wanted, since every pre-upgrade week is a week of history that will never have identities.
Snapshot retention interacts with all of it, and the interaction reshapes a familiar setting. Changelog computation and history walks read at snapshots, so a consumer's reach backward is bounded by expiration policy, and the tiny-commits guidance to expire aggressively now carries a lineage clause: retention must cover the slowest changelog consumer's maximum lag, watermark to current, with margin, or a delayed consumer wakes to find its "old" snapshot expired and its delta uncomputable except as a full refresh. The identities themselves survive expiration fine, they live in the current data, and the ability to diff against a departed snapshot does not. Retention stops being purely a storage-cost dial and becomes part of the changelog contract, sized per table against its consumers, one more example of v3 features promoting maintenance settings into semantics.
Operational Guidance#
The condensed practice list for teams adopting lineage deliberately.
Upgrade the change-heavy tables first, and on purpose. The CDC mirrors, the upsert targets, the tables whose downstream consumers rebuild rather than increment, these convert lineage into value immediately, and their pre-upgrade history matters least since their state churns anyway.
Probe your writers once per engine. Run a known small update through each writer that touches a shared v3 table and inspect the result with the changelog recipe: one UPDATE row is a pass, a DELETE-INSERT pair is a severed chain and a bug report. An hour of probing per engine buys certainty the capability matrices cannot.
Put lineage requirements in the maintenance contract. The compactor materializes both fields, verified on its outputs with a metadata query, and the verification runs after tool upgrades, because maintenance tooling is where lineage breaks quietly and where the fix is a configuration rather than an incident if caught in staging.
Build the watermark pattern as shared infrastructure. One small library or dbt macro, snapshot-pinned reads, sequence watermarks, the full-outer-join recipe, serves every incremental consumer in the organization, and centralizing it means the edge cases, branch boundaries, pre-upgrade nulls, get handled once.
And record the upgrade date per table beside the retention policy, because "how far back does our changelog go" will be asked, and the answer is a lookup for teams that wrote it down and an investigation for teams that did not.
A closing note on sequencing all of this against the rest of a v3 adoption: lineage rides free on the upgrade you were already planning for deletion vectors or Variant, requires no enablement, and starts accruing history from the first post-upgrade commit, which makes it the rare feature whose adoption checklist is mostly "upgrade, probe your writers, and start the clock." The five items above are an afternoon per important table, and the afternoon buys a changelog whose value compounds with every week it runs, the strongest possible argument for spending it early rather than retrofitting it the quarter someone asks for incremental refresh.
Where This Is Heading#
The nearest milestone is packaging. The changelog recipe, the watermark pattern, the history walk, all of them are today the practitioner's SQL and tomorrow the engine's function, and the packaging wave is already visible in changelog table functions, incremental read APIs, and view-maintenance machinery that consume lineage underneath a friendlier surface. Packaging matters beyond convenience: it standardizes edge-case handling, the branch boundaries, the null-lineage history, the writer-conformance checks, and the recipes in this article are best understood as the semantics those functions must honor, useful directly now and as an evaluation rubric for the conveniences later.
Three trajectories extend the story beyond packaging. Inside Iceberg, the v4 direction treats change as a first-class product, cheaper to record through the metadata redesign, finer-grained through the column-update proposals, and lineage is the identity layer all of it assumes, the difference between "these bytes changed" and "these rows, with these histories, changed." Across formats, the convergence is already visible, row tracking on the neighboring format serving the same incremental machinery, every Iceberg v3 table carrying it by mandate, and a shared identity vocabulary at the metadata layer is one more place where the format war's end looks like engineering agreement. And above the formats, the consumers are assembling: incremental view maintenance moving from platform feature toward table-format expectation, feature platforms and vector indexes learning to refresh from changelogs, agent frameworks learning to cite row-grain provenance, each one a bet that the substrate will keep answering "what changed" precisely. The bets are correct. The substrate now does.
Conclusion#
Row lineage gives Iceberg tables the primitive they always lacked: rows with names, and names with dates attached. The mechanism is spare, two reserved fields, one table counter, an inheritance rule that makes appends free, materialization exactly where derivation fails, and the consequences are anything but, a changelog in two columns, CDC without sidecars, incremental computation with precise deltas, audits that cite the format instead of the guess, and an AI data layer whose training sets, features, and agents can all say which row versions they touched. The fine print is real, engine unevenness, the identity-versus-key line, history starting at the upgrade, and none of it dims the trajectory. Deletion vectors made v3's changes cheap, Variant made its data flexible, and row lineage, the feature the roundups bury, made its changes mean something. Buried third, remembered first, and worth adopting before the workloads that depend on it arrive asking where the history went. The tables you upgrade this quarter start writing their biographies immediately, and biographies, unlike features, cannot be backfilled.
Keep Going#
If this piece was useful, I have written a lot more on Apache Iceberg and lakehouse architecture. Apache Iceberg: The Definitive Guide, which I co-authored for O'Reilly, covers snapshots, sequence numbers, and the metadata machinery lineage builds on. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.
