Cross-posted. This article's canonical home is iceberglakehouse.com.

AI applications are messy data producers. They create prompts, completions, tool calls, retrieval traces, ranking signals, evaluation scores, safety annotations, user feedback, latency metrics, and error payloads. Some of that data is stable enough to deserve clean columns on day one. Much of it is not.

That tension is exactly why semi-structured data support matters in the lakehouse. Teams want the freedom to capture evolving AI payloads without stopping every release for a schema migration. They also need the discipline to turn the parts that matter into governed, queryable, trusted data products. If everything stays as raw JSON forever, analytics becomes guesswork. If everything has to be modeled before it lands, AI teams move faster than the data platform can keep up.

The Iceberg v3 discussion around richer type support, including variant-style handling for semi-structured values, belongs in that practical middle ground. The important idea is not that one type magically solves AI data. The important idea is that open table formats need to represent messy machine-generated data in a way that stays portable across engines, catalogs, and governance systems.

I am using the phrase "variant-style" deliberately. Exact feature language and support varies by specification version and engine implementation, so final production claims should be checked against the current Apache Iceberg spec and the specific engines in use. The architectural need is clear either way: AI JSON payloads need a home that is flexible at ingestion and disciplined at consumption.

That is where the lakehouse becomes interesting.

Why AI Payloads Do Not Fit Neatly into Traditional Tables

Relational modeling works best when the shape of the data is understood. A customer table has customers. An orders table has orders. A product catalog has products. The details can change, but the basic entity usually has a stable meaning.

AI telemetry is different. A single request can include an input prompt, retrieved chunks, tool definitions, tool arguments, model settings, policy decisions, token counts, latency measurements, model output, citations, follow-up steps, and human feedback. A month later, the application might add a new retrieval strategy, a new evaluator, a new safety classification, or a new set of tool-call attributes.

If the data platform requires a fully normalized schema before the application can land that payload, teams will either slow down or route around the platform. If the platform accepts the whole payload as opaque text, teams will have trouble analyzing it later. Neither outcome is healthy.

This is why semi-structured support has become a core analytical requirement. AI teams need to preserve the original payload because it is often the evidence trail for debugging and evaluation. Data teams need to extract and govern the fields that become business-critical. Security teams need to mask or restrict sensitive fragments. Compliance teams need to understand lineage. Product teams need to compare behavior across model versions.

The problem is not "JSON is bad." The problem is unmanaged JSON. A lakehouse should let teams land flexible payloads while creating a path toward structure, quality, and trust.

What Variant-Style Typing Is Trying to Solve

A variant-style type is useful when a column may contain semi-structured values whose exact shape can vary from row to row. Instead of forcing every nested attribute into a fixed relational schema immediately, the table can preserve a structured value that engines may inspect, project, and transform over time.

For AI payloads, that matters because the outer event may be stable while the inner details evolve. A table might have stable columns such as event_id, tenant_id, user_id, application_id, model_id, event_timestamp, and event_type. Alongside those columns, it may store a semi-structured payload with request-specific details.

For example, an agent tool-call event might include:

• the tool name
• input arguments
• output summary
• execution status
• latency
• retry count
• approval state
• policy checks
• error details

Some of those fields may later become first-class columns. Others may remain nested because they are rarely queried or change often. A variant-style column gives teams a way to preserve the full event while gradually promoting important fields.

That gradual promotion is the key design pattern. The goal is not to leave the data raw forever. The goal is to avoid making the ingestion schema a bottleneck while still giving the organization a clear path toward governed analytics.

Why Open Table Formats Matter for Semi-Structured AI Data

Many engines and warehouses have their own ways to store and query JSON. Those features are useful, but they can create portability problems. If an organization stores critical AI telemetry in a proprietary semi-structured type that only one platform understands well, the data becomes less open even if it sits on object storage.

Open table formats push against that risk. If Iceberg can represent semi-structured data in a standard way, multiple engines can build support around the same table semantics. That does not mean every engine will perform identically or expose identical SQL syntax. It does mean the storage and metadata layer has a better chance of remaining portable.

For AI systems, portability is not an abstract preference. Model evaluation, product analytics, governance, security, and operations often use different tools. Data scientists may inspect payloads in notebooks. Analytics engineers may build semantic views. Security teams may run audits. Product teams may use BI. AI agents may query approved metrics. The same underlying data should be available to those workflows without requiring brittle exports.

This is one of the reasons I keep coming back to Iceberg as an important foundation for AI-era data architecture. The more machine-generated data a company creates, the more it needs a durable and open way to store that data. Closed formats and one-engine assumptions get more expensive as the number of consumers grows.

A Practical Table Design for AI JSON Events

The worst design is usually the all-or-nothing design. One extreme is a table with hundreds of columns that tries to model every possible payload detail. The other extreme is a table with a timestamp and a giant raw JSON blob. A better design uses layers.

At the landing layer, keep the event envelope stable. Include identifiers, timestamps, tenant or workspace scope, application name, event type, model or tool identifier, and version fields. These columns make the table partitionable, filterable, governable, and joinable.

Then store the original payload in a semi-structured column. This preserves the evidence. If a later investigation needs the exact tool arguments or response metadata, the data is still there.

Next, promote high-value fields into typed columns or curated views. These are fields that appear in common filters, joins, dashboards, quality checks, or semantic metrics. Examples include latency, token count, evaluation score, tool status, model version, retrieval strategy, and safety outcome.

Finally, build semantic views for business-facing consumption. Most users should not have to understand the raw event structure. They should work with clear concepts such as model success rate, average tool latency, grounded answer rate, escalation rate, cost per task, or agent completion rate.

This layered approach keeps ingestion flexible and analytics disciplined.

The Difference Between Flexible and Sloppy

Flexibility is not the same as looseness. A semi-structured payload column can become a dumping ground if the team does not apply governance.

The first control is documentation. Each event type should have an owner and a description. If an event represents a model response, tool call, evaluation result, or user feedback action, that meaning should be clear. The payload may be flexible, but the event itself should not be mysterious.

The second control is versioning. AI payloads change because applications change. Include schema or payload version fields so consumers can understand how interpretation differs across time. Without versioning, downstream logic becomes full of fragile assumptions.

The third control is data quality. Even flexible payloads need tests. Required envelope fields should be present. Timestamps should be valid. Tenant IDs should map to known tenants. Event types should match an allowed set. Sensitive fields should be detected and handled according to policy.

The fourth control is access. Semi-structured payloads often contain sensitive data. A prompt may include personal information. A tool argument may include customer details. A model response may contain content that should not be broadly visible. Column-level access alone may not be enough if sensitive values are nested inside a variant-style payload. Teams need masking, filtering, and safe views.

The fifth control is promotion discipline. When a nested field becomes important, promote it intentionally. Give it a type, name, owner, and definition. Add tests. Decide whether it belongs as a physical column, a derived view field, or a semantic metric.

That is how a lakehouse can stay flexible without becoming a junk drawer.

How This Supports Agentic Analytics

Agentic analytics changes the consumption pattern. A human analyst might inspect a table carefully before writing a query. An AI agent may inspect metadata, generate a query, evaluate a result, and call another tool within seconds. That speed makes semantic clarity more important, not less.

If the agent sees only a raw payload column, it has to infer too much. It may choose the wrong nested field. It may miss a version change. It may misunderstand a status value. It may summarize incomplete data as if it were complete. These are not model problems alone. They are data product problems.

A better design gives the agent multiple levels of context:

• raw payload access for approved debugging use cases
• curated fields for common analytical filters
• semantic metrics for business questions
• lineage and freshness metadata
• policy-aware views
• documentation for event types and versions

This lets the agent operate with constraints. It can answer common questions through trusted semantic assets. It can ask for raw payload details only when the user has permission and the task requires it. It can explain which version of an event definition it used.

In other words, variant-style data support helps capture the complexity of AI systems, but the semantic layer helps make that complexity usable.

The Query Performance Question

Semi-structured data can be expensive to query if every analysis scans and parses large nested payloads. That is why table design and workload management matter.

A common mistake is to land a large JSON payload and then repeatedly extract the same fields at query time. That may be acceptable during exploration, but it becomes costly for production metrics. If a field is queried constantly, promote it. If a field is used for partition pruning or common filters, do not bury it deep inside a payload. If a field powers dashboards or agent tools, give it a stable definition.

Iceberg helps because table metadata can support pruning, planning, schema evolution, and snapshot-aware access. But Iceberg is not a substitute for good modeling. The table format gives you the foundation. The data team still has to decide which fields deserve structure.

This is also where query acceleration and autonomous optimization become important. AI telemetry can grow quickly. Agent traces are verbose. Evaluation data multiplies with every model, prompt, and test set. If the platform cannot keep common queries fast, teams will create extracts and side tables. Those side tables often become new governance problems.

A Dremio-style lakehouse approach becomes compelling here because it emphasizes open data, high-performance query access, semantic consistency, and acceleration without requiring every workload to be copied into a single warehouse. The architecture nudges teams toward making the lakehouse itself fast and useful.

Governance for Prompts, Tool Calls, and Model Outputs

AI payloads are not ordinary application logs. They may contain user-entered text, internal reasoning traces, retrieved documents, generated summaries, tool arguments, and decisions that affect customers or operations. That raises the governance bar.

Prompt data should be classified. Some prompts may be safe for broad product analytics. Others may contain personal data, contractual details, medical references, or confidential business information. Tool-call arguments may be even more sensitive because they can contain structured operational data. Model outputs may need moderation or retention rules.

The table design should account for this from the start. Sensitive payload sections may need to be separated, masked, tokenized, or excluded from certain views. Access policies should distinguish between aggregate analytics and record-level inspection. A product manager may need completion rates by feature. A security investigator may need raw request details. Those are different access patterns.

Lineage matters too. If a model output used retrieved documents, the system should be able to trace which sources contributed. If an evaluation score changed because the rubric changed, that should be visible. If a semantic metric excludes certain event types, that definition should be documented.

Variant-style data makes it possible to preserve rich detail. Governance makes it safe to use.

Why This Is Not Just a Storage Conversation

It is easy to treat semi-structured support as a storage feature. Can the table hold nested data or not? That question matters, but it is too narrow.

The real question is whether the data platform can support the full lifecycle:

• capture the original AI event
• preserve table history
• evolve the schema
• promote useful fields
• secure sensitive payload fragments
• expose trusted metrics
• support multiple query engines
• provide lineage and auditability
• serve agents and humans from the same governed foundation

That lifecycle is where the lakehouse architecture becomes more valuable. Object storage provides scale. Iceberg provides table semantics. Catalogs provide discovery and governance hooks. Query engines provide access and performance. Semantic layers provide business meaning. Agent interfaces provide new modes of interaction.

No single layer is enough by itself.

A Dremio-Positive Reading of the Trend

The most durable conclusion is that AI workloads make open lakehouse architecture more important, not less. AI teams will create huge amounts of semi-structured data. Different teams will need to inspect that data through different tools. Some queries will be exploratory. Some will power dashboards. Some will feed evaluations. Some will be called by agents. Some will be part of compliance and incident review.

If that data is locked inside a single proprietary system, the organization loses flexibility. If it is dumped into raw object storage with no table semantics, the organization loses trust. If it is modeled too rigidly too early, the organization loses speed.

The attractive middle path is open, governed, and iterative. Land flexible payloads. Preserve original evidence. Promote important fields. Build semantic assets. Keep the table format open. Let multiple engines participate. Make performance good enough that teams do not need to create uncontrolled copies.

That is why the Dremio Lakehouse direction fits the moment. It is not because every semi-structured question must be answered by one engine. It is because the architectural conclusion points toward open tables, query federation, semantic meaning, and governed acceleration. Those are exactly the pieces organizations need as AI data grows.

Practical Design Recommendations

Start with the event taxonomy. Before debating types, decide what events the AI application emits. Separate user request events, model response events, retrieval events, tool-call events, evaluation events, feedback events, and operational error events. Each event should have an owner and a purpose.

Define a stable envelope. Even if the payload varies, every event should include identifiers, timestamps, tenant scope, application version, model or tool identity, event type, and payload version. This makes the table operationally useful.

Preserve the original payload. Debugging AI systems often requires going back to the exact request and response context. Do not throw that away too early.

Promote fields based on usage. If a nested field becomes a common filter, join key, metric input, or governance control, promote it into a typed column or curated view.

Create semantic metrics for common questions. Examples include agent success rate, tool failure rate, grounded response rate, average response cost, average retrieval count, escalation rate, and policy intervention rate.

Classify sensitive fields. Do not assume all nested payload content is safe. Prompts and tool arguments deserve special attention.

Test across engines. If portability matters, validate how the chosen engines read, write, and query semi-structured fields. Open table format support still depends on implementation maturity.

Track version changes. Payload schemas will evolve. Make that evolution visible.

Avoid performance surprises. Do not repeatedly parse large payloads for high-traffic metrics. Promote, materialize, or accelerate the fields that matter.

Expose only trusted assets to agents. Agents should use semantic views and governed tools by default, not raw payloads.

The Direction I Would Bet On

AI data will not become simpler. Agents will create more traces, more tool calls, more evaluations, and more operational events. The data will be valuable because it explains how AI systems behave. It will also be risky because it may contain sensitive context and fast-changing structure.

The winning architecture will not force every payload into a rigid schema before landing. It also will not leave everything as opaque JSON. It will support a gradual path from flexible capture to governed analytical use.

That is the real promise of variant-style semi-structured support in open table formats. It helps the lakehouse absorb the shape of AI data without giving up the discipline required for analytics, governance, and agentic workflows.

For practitioners, the next step is practical: inventory the AI events your systems already emit, define the stable envelope, decide what belongs in raw payloads, and identify which fields should become trusted columns or metrics. The sooner that discipline is in place, the easier it becomes to build reliable AI analytics on top.

One small habit helps more than teams expect: review promoted fields every release cycle. If a nested attribute appears in repeated investigations, dashboards, evaluation jobs, or agent prompts, treat that as a signal that it may deserve a real definition. That turns schema evolution into a product habit rather than a cleanup project.

For more of my thinking on data lakehouse and AI architecture, explore my books at books.alexmerced.com. To try a modern Agentic Lakehouse experience, visit dremio.com/get-started.