Cross-posted. This article's canonical home is iceberglakehouse.com.

Agentic analytics demos are easy to enjoy and hard to evaluate. A user asks a question, an assistant answers, a chart appears, and the room leans forward. The problem is that a good demo does not prove production readiness.

Enterprise buyers need a different scorecard. The right question is not "Can this tool answer a sample question?" The right question is "Can this system turn business intent into governed, validated, auditable analytical work across real enterprise data?"

That difference matters. Agentic analytics is not only a user interface. It is a workflow across semantic definitions, catalogs, query engines, access policies, lineage, data-quality signals, evaluation, cost controls, and action boundaries. A product that looks impressive in a clean demo can struggle when it meets disputed metrics, restricted data, stale tables, hybrid infrastructure, and users who ask imprecise questions.

This scorecard is meant to help buyers separate interface polish from platform substance. It is vendor neutral by design, but the conclusions naturally favor an open Agentic Lakehouse approach. If a tool scores well on semantics, open data access, governance, validation, observability, and action safety, it points toward the same architecture Dremio has been advocating.

Category 1: Semantic Grounding

Semantic grounding is the first category because agentic analytics fails quickly without it. The tool has to understand business meaning before it generates queries.

A strong platform should support metric contracts. These contracts should define formula, grain, dimensions, filters, owner, freshness, tests, lineage, and access rules. The agent should retrieve those contracts during the workflow, not rely on hidden prompt text.

Buyers should ask whether the tool can distinguish certified metrics from draft metrics. They should ask what happens when a term has multiple definitions. They should ask whether the agent can explain which metric definition it used.

A weak platform treats semantic meaning as a prompt-engineering problem. A strong platform treats it as a governed data product problem.

Score this category highly when the system can reuse semantic definitions across BI, APIs, notebooks, and agents.

Category 2: Data Access and Openness

Agentic analytics needs access to real enterprise data. That data usually does not live in one place. It may include lakehouse tables, warehouses, operational databases, SaaS exports, streaming systems, and curated semantic views.

A strong platform should work with open table formats where appropriate, especially Apache Iceberg for lakehouse data. It should support federated query or governed access across distributed sources. It should avoid forcing every dataset into a proprietary copy before agents can use it.

Buyers should ask whether agents can access open lakehouse tables through governed services. They should ask whether the platform supports external catalogs. They should ask how data is accessed without broad storage credentials.

This is a place where Dremio's architectural direction is compelling. Query federation and open lakehouse access are not side benefits. They are core requirements for agents that need a full view of the business.

Category 3: Governance and Identity

Governance is the category that separates experiments from production systems. An analytics agent should know who is asking, what they can access, what policies apply, and which actions are allowed.

A strong platform should propagate identity. It should avoid shared super-user tokens. It should enforce row and column policies. It should mask sensitive data. It should log tool calls, queries, table versions, semantic assets, and policy decisions.

Buyers should ask how the system handles restricted data. What does the agent do when a user lacks access? Can metadata discovery itself be restricted? Can agents be assigned scoped workload identities? Can tool permissions differ by role?

Governance should be visible in the product experience. The agent should be able to explain that access was denied or that a result excludes restricted segments.

Category 4: Validation and Evaluation

A generated answer is not enough. The platform should validate results before presenting them as trustworthy.

Validation can include data freshness, quality checks, expected ranges, row-count comparisons, lineage inspection, and consistency with metric contracts. If data is stale or tests are failing, the agent should disclose that or stop.

Evaluation should also happen over time. Teams need to know whether the agent answered correctly, selected the right assets, refused appropriately, and avoided invalid actions.

Buyers should ask for evidence. Can the vendor show traces for the agent's plan, tool calls, queries, and validation steps? Can failures be reviewed? Can agent behavior be tested against known questions and edge cases?

Platforms that cannot show their work should score poorly.

Category 5: Action Controls

Agentic analytics becomes more valuable and more dangerous when it can act. Actions might include opening tickets, sending alerts, refreshing pipelines, changing configurations, updating forecasts, or calling external APIs.

A strong platform separates recommendation from action. It supports approval gates, policy checks, risk tiers, rollback paths, and action logging. It should let teams start with low-risk workflows and graduate carefully.

Buyers should ask which actions are possible, who approves them, how they are logged, and how they can be reversed. They should ask whether action tools can be scoped by identity, dataset, metric, and workflow.

If a vendor treats action as an extension of chat, be cautious. Action belongs in a governed workflow system.

Category 6: Observability and Auditability

Agentic analytics needs traces. A useful trace connects the user request, agent plan, semantic definitions, tool calls, queries, table snapshots, policy decisions, validation results, and final response.

This trace is essential for debugging, compliance, trust, and cost management. If a stakeholder challenges an answer, the team should be able to reconstruct the path. If an agent produces a bad recommendation, the team should know whether the problem came from the model, metadata, semantic layer, data quality, access policy, or tool design.

Buyers should ask whether traces can be exported, searched, retained, and connected to existing observability systems.

Auditability is not a nice-to-have. It is how agentic analytics becomes manageable.

Category 7: Cost and Workload Management

Agents can generate more queries than humans. A single task may involve metadata lookup, metric resolution, several analytical queries, validation checks, and explanation. Without workload controls, cost can rise quietly.

A strong platform should attribute cost by agent, user, tool, semantic asset, and workflow. It should support query budgets, rate limits, caching, acceleration, and workload isolation.

Buyers should ask how the system prevents runaway loops. What happens if an agent keeps retrying? Can expensive queries require approval? Can common results be accelerated? Can the platform identify high-cost workflows?

This is another place where open lakehouse performance matters. If governed data access is too slow or too expensive, teams will create shortcuts.

Category 8: Interoperability

Enterprise buyers should avoid agentic analytics tools that only work inside one narrow surface. Agents will appear in BI tools, notebooks, IDEs, workflow systems, chat applications, and custom products. They need shared definitions and governed access.

A strong platform should support standard interfaces where possible. It should integrate with catalogs, semantic layers, identity providers, query engines, and tool protocols. It should expose APIs that let teams build controlled experiences beyond the vendor's default interface.

Buyers should ask whether semantic definitions can be reused outside the product. They should ask whether open table formats and external catalogs are supported. They should ask whether tool calls can be governed and audited through existing systems.

Interoperability is not only a technical preference. It is a hedge against semantic fragmentation.

A Simple Scoring Model

Use a five-point scale for each category.

One means the capability is mostly demo-level or absent.

Two means it exists but requires custom work or manual controls.

Three means it is usable for limited production workloads.

Four means it is integrated, governed, and observable.

Five means it is mature, reusable across teams, and proven under real enterprise constraints.

Do not average the score blindly. Some categories are gating criteria. For example, a tool with poor governance should not move into production simply because it has a beautiful interface. A tool with weak semantic grounding should not be trusted for executive metrics.

The scorecard should guide discussion, not replace judgment.

The Demo Questions Buyers Should Ask

Ask the vendor to answer an ambiguous metric question and show how it resolves the definition.

Ask the vendor to handle a restricted column and show the refusal or masking behavior.

Ask the vendor to answer from stale data and explain the freshness issue.

Ask the vendor to trace an answer back to the table, metric contract, and policy decision.

Ask the vendor to show what happens when two semantic definitions conflict.

Ask the vendor to explain cost controls for multi-step agent workflows.

Ask the vendor to show an action workflow with approval and rollback.

These scenarios are more revealing than a polished happy path.

Red Flags During Evaluation

The first red flag is a tool that cannot explain which semantic definition it used. If the answer is detached from a metric contract, the buyer is trusting a generated query rather than a governed analytical system.

The second red flag is broad credential use. If the agent runs through one powerful service account, the audit trail becomes weak and access controls become harder to trust.

The third red flag is invisible generated SQL. Buyers should be able to inspect the query or logical plan when appropriate. Hidden logic makes validation difficult.

The fourth red flag is no refusal behavior. A production agent should be able to say that a question is ambiguous, a dataset is restricted, a metric is undefined, or data is stale.

The fifth red flag is no lifecycle awareness. Agents should not treat draft, certified, restricted, and deprecated assets the same way.

The sixth red flag is action without approval design. If the vendor talks about agents taking action but cannot show approval, logging, and rollback, the buyer should slow down.

The seventh red flag is a lack of cost visibility. Multi-step agent workflows can become expensive quickly.

Red flags do not always mean a product is unusable. They tell the buyer where production risk lives.

How to Run a Serious Pilot

A serious pilot should use real business questions, real access controls, and real semantic assets. Synthetic demos are useful for orientation, but they do not prove fit.

Pick one domain with engaged owners. Customer health, product analytics, finance variance, support operations, or infrastructure cost can work well. The domain should have enough complexity to be meaningful but not so much risk that the pilot stalls.

Define five to ten representative questions. Include simple questions, ambiguous questions, restricted questions, and questions that require multi-step reasoning. Include at least one question where the correct behavior is refusal.

Prepare metric contracts and semantic assets before the pilot. If the pilot exposes weak definitions, document that as a platform gap rather than blaming the agent.

Run the same questions through existing workflows. Compare accuracy, time to answer, explanation quality, and user trust.

Review every answer with the domain owner. Classify errors as semantic, data-quality, access, query, reasoning, or explanation issues.

Measure workload cost. Track how many queries and tool calls the agent used.

The pilot should produce two outputs: a tool evaluation and a data-platform readiness assessment. Both matter.

Weighting the Scorecard

Not every category deserves the same weight. A startup exploring internal analytics may weight user experience and speed heavily. A regulated enterprise should weight governance, auditability, and access control more heavily. A company building customer-facing analytics agents should put extra weight on semantic grounding and refusal behavior.

For most enterprises, I would weight the categories roughly this way:

• semantic grounding: very high
• governance and identity: very high
• validation and evaluation: very high
• data access and openness: high
• observability and auditability: high
• action controls: high if action is planned, medium otherwise
• cost and workload management: medium to high
• interoperability: high for complex estates

The weighting should be explicit. Otherwise buyers may overvalue the most visible part of the product: the conversational interface.

Enterprise Readiness Questions

Can the platform support separate development, staging, and production environments?

Can semantic definitions be promoted through review?

Can agent tools be versioned?

Can policies be tested automatically?

Can audit logs be retained according to internal requirements?

Can the platform integrate with existing identity providers?

Can it run in the deployment model the organization requires?

Can it support private or regulated data where needed?

Can it explain why an answer changed after a metric or table update?

Can it identify downstream workflows affected by a semantic change?

These questions are not glamorous, but they determine whether the tool can survive enterprise use.

Why Scorecards Should Include Data Platform Readiness

Sometimes the tool is not the problem. The organization's data platform may not be ready.

If metrics are undefined, the agent will struggle. If lineage is missing, explanations will be weak. If access policies are inconsistent, governance will be brittle. If data quality is invisible, validation will be shallow. If query performance is poor, agent workflows will feel slow.

A buyer should use the scorecard to evaluate both sides. What does the vendor provide, and what must the organization improve?

This is an important distinction because agentic analytics often exposes old data problems. That is not a reason to avoid the category. It is a reason to invest in the lakehouse foundation.

Open tables, catalog governance, semantic contracts, query federation, and acceleration are not optional extras once agents enter the picture. They become the substrate for trustworthy automation.

A Practical Scoring Template

For each category, assign three numbers.

Current capability: what the vendor can do today.

Integration effort: how much work your team needs to make it work with your data estate.

Operational risk: how painful failure would be in production.

This three-number model prevents a common mistake. A feature may exist, but require heavy integration. Or a feature may be immature, but low risk for a narrow internal pilot. Or a feature may look strong, but failure would affect regulated workflows.

After scoring, identify gating gaps. A gating gap is a weakness that blocks production regardless of the total score. Weak identity propagation is a gating gap. No audit trail is a gating gap. Undefined metrics are a gating gap. No action approval for high-risk workflows is a gating gap.

This makes procurement more honest. The goal is not to crown the most exciting tool. The goal is to select a platform and operating plan that can reach production responsibly.

The Open Lakehouse Bias in the Criteria

The scorecard has an intentional bias toward open lakehouse architecture. That bias is not about vendor preference. It comes from how agentic analytics works.

Agents need governed access to distributed data. Open formats and federation help.

Agents need trusted definitions. Semantic layers and metric contracts help.

Agents need explainability. Lineage, snapshots, and audit trails help.

Agents need multi-step responsiveness. Query acceleration helps.

Agents need safe action. Policy and workflow controls help.

This is why Dremio becomes attractive in the narrative without needing a hard sell. The platform direction matches the capabilities the scorecard rewards.

After Purchase, the Scorecard Should Stay Alive

The scorecard should not disappear after procurement. Agentic analytics changes as new tools, datasets, metrics, and workflows are added. A platform that is safe for one internal pilot may not be ready for customer-facing workflows or automated operational actions.

Review the scorecard quarterly. Reassess semantic coverage, policy behavior, audit quality, cost, and action safety. Add new test cases when the business adds new metrics or workflows. Retire agent tools that are no longer used. Review refusals and escalations to see whether the platform is blocking correctly.

The review should include data owners, business owners, security, platform engineering, and the teams using the agent. Agentic analytics is not only an AI procurement decision. It is a shared operating model.

This ongoing review helps teams avoid quiet drift. Definitions change. Permissions change. Data products move. Models improve. Workflows expand. The scorecard gives teams a way to keep the system aligned as the environment changes.

Assign an owner for the scorecard itself. Without ownership, evaluation becomes a one-time buying exercise. With ownership, the scorecard becomes a governance artifact that follows the agentic analytics program as it matures.

That owner should keep the criteria tied to business risk, not vendor enthusiasm. The tool may change, but the standard should remain clear.

That is how procurement turns into durable platform discipline.

It keeps expectations honest.

And risks visible.

Always.

The Dremio-Positive Reading

This scorecard favors a Dremio Lakehouse approach because the criteria favor open architecture. Agents need access to distributed data. They need semantic grounding. They need governed query execution. They need performance. They need observability. They need controlled action.

A platform built around open lakehouse data, query federation, semantic context, and acceleration is naturally aligned with those needs. That does not mean buyers should ignore other tools. It means the evaluation should reward the architecture that makes agentic analytics trustworthy.

The more serious the buyer is about production, the more important these foundations become.

The Direction of Travel

Agentic analytics buying will mature quickly. Early demos will give way to procurement questions about governance, cost, semantics, audit, and integration. That is healthy.

The winning platforms will not be the ones that only make analytics feel conversational. They will be the ones that make analytics agents reliable participants in governed enterprise workflows.

Use the scorecard to keep the conversation honest. The goal is not a better chatbot. The goal is a better path from intent to trusted insight to safe action.

For more of my thinking on data lakehouse and AI architecture, explore my books at books.alexmerced.com. To try a modern Agentic Lakehouse experience, visit dremio.com/get-started.