Cross-posted. This article's canonical home is iceberglakehouse.com. By Alex Merced, Head of Developer Relations at Dremio

The most important discovery of the agent era fits in one sentence: most AI failures are context failures, not model failures. When your assistant gives a generic answer, forgets what you told it last week, invents a metric definition, or confidently applies last quarter's policy, the model underneath was usually working fine. What failed was the pipeline that decides what the model knows at the moment it answers.

The industry has a name for the discipline that fixes this: context engineering, the successor to prompt engineering, concerned not with what words you use but with what configuration of information reaches the model at decision time. And within context engineering, one distinction matters more than any other, because it splits the problem into two genuinely different games: personal context, the accumulated knowledge that makes an AI useful to one specific human, and shared context, the governed knowledge that makes AI trustworthy across an organization.

I live on both sides of that split. Personally, I run an intensive AI-assisted content operation, weekly newsletters, long-form articles, books, built on carefully engineered personal context: style rules, source pipelines, project files, and accumulated preferences that make my tools an extension of how I work. Professionally, I spend my days in the enterprise version of the problem, where the question is how thousands of employees and their agents can draw on organizational knowledge without shredding permissions, provenance, or truth. The two games share vocabulary and diverge on almost everything else: what works, what breaks, and what the failure modes cost.

This article is the deep dive into both. The foundations of context and memory, stated plainly. The major approaches to personal context, files, built-in memory, memory services, personal knowledge bases, with honest pros and cons. The enterprise approaches to shared context, governed retrieval, semantic layers, catalogs, context platforms, with equally honest challenges. The seam where the two meet, which is where the interesting design problems live. And frameworks for choosing, whether you are optimizing your own workflow or architecting for ten thousand employees.

Foundations: Context, Memory, and Why Both Fail

Four concepts underpin everything that follows, so let me establish them without jargon.

The context window is a scarce resource. Every model call happens inside a window: the finite text the model can attend to while generating its answer. Everything the model "knows" in that moment, instructions, retrieved documents, conversation history, tool outputs, memories, competes for that space. Windows have grown enormous, and scarcity persists anyway, for a subtle reason: model attention degrades as windows fill. Stuff two hundred pages of marginally relevant material into a call and the model performs worse than with five carefully chosen pages. Practitioners call the degradation context rot, and it converts context engineering from a stuffing problem into a curation problem. More is not better. Right is better.

Memory is what persists between calls. Models are stateless: nothing survives from one call to the next unless some system carries it forward. Memory systems are that carrier, and they come in recognizable types borrowed loosely from cognitive science: episodic memory, records of what happened in past interactions, semantic memory, distilled facts and preferences, and procedural memory, learned ways of doing things. The relationship to context is mechanical: memory lives outside the window in some store, and at answer time, a retrieval step selects which memories earn their place inside the window. Memory without good retrieval is a diary nobody reads. Retrieval without good memory is a search over nothing.

Context assembly is the real product. Every serious agent system runs an assembly pipeline before each model call: gather the standing instructions, retrieve relevant knowledge, select relevant memories, include recent conversation, attach tool results, and compress or truncate to fit. Every approach in this article, personal or enterprise, is ultimately a different answer to who runs this pipeline, what stores feed it, and what rules govern the selection.

And context fails in characteristic ways. Context rot, already introduced: quality degrading as volume grows. Context poisoning: wrong or malicious information entering the stores and then being retrieved with the same confidence as truth, which in enterprise settings escalates from bug to attack surface, since anything an agent reads is a potential injection vector. Context staleness: yesterday's truth served today, the policy that changed, the metric that was redefined. And context leakage: information crossing boundaries it should not, the personal note surfacing in a shared setting, the executive document retrieved for an intern's query. Hold these four failures, because every architecture below is an attempt to manage some subset of them, and no architecture manages all four for free.

Personal Context: The Approaches, Honestly Compared

Start with the individual, because personal context is where most people first feel the problem: the assistant that would be twice as useful if it just remembered how you work. Four approaches dominate in 2026, each with a distinct philosophy.

Approach one: files as context. The oldest trick and still the strongest foundation: write your context down in plain files that get loaded into every relevant session. The developer world converged on conventions for exactly this, instruction files like CLAUDE.md and the AGENTS.md convention that repositories now carry to guide AI coding assistants, and the same pattern generalizes: project briefs, style guides, standing preferences, all as versioned text.

The pros are the pros of files everywhere: total transparency, you can read exactly what your agent knows, total control, you edit it deliberately, portability, the file moves to any tool that reads text, and versionability, your context has a Git history. My own operation runs on this approach: my newsletters are produced under explicit style-rule files, banned vocabulary, structural conventions, sourcing requirements, that function as durable procedural context, and the quality difference between sessions with and without those files loaded is not subtle. The cons are equally structural: files are manual, they capture what you remembered to write down rather than what actually happened, they go stale unless you maintain them, and they do not scale past what you can curate by hand. Files are deliberate memory. Nothing that happens automatically lands in them.

Approach two: built-in memory features. The major assistants now ship automatic memory: the system observes your conversations, extracts facts and preferences, and quietly includes relevant ones in future sessions. You mentioned once that you prefer concise answers, that you use Python, that your newsletter goes out Thursdays, and the assistant just knows, thereafter.

The pro is the effortlessness: memory accrues from use, capturing things you would never have thought to write down. The cons deserve more attention than they get. Opacity: you often cannot see exactly what was stored or why a memory surfaced, which makes failures hard to debug. Drift: automatic extraction stores misunderstandings alongside truths, and a wrong memory is worse than no memory because it arrives with confidence. Staleness: preferences change and stored facts linger. And above all, lock-in: memory accumulated inside one vendor's product does not travel. Your assistant's understanding of you is an asset, and with built-in memory, it is an asset you do not hold. The mature posture is to treat built-in memory as convenience, audit it periodically where the product allows, and keep anything you would grieve losing in files you own.

Approach three: dedicated memory services. A whole ecosystem now sells memory as infrastructure: services like Mem0, Supermemory, Membase, Zep, and a rapidly growing open source field, offering APIs that extract, store, compress, and retrieve memories across any agent that integrates them. The engineering is genuinely impressive: compression engines that distill interaction history into optimized representations claim token reductions up to 80 percent while preserving fidelity, knowledge-graph backends capture relationships rather than just facts, and the universal memory pitch, one memory layer that follows you across every AI tool you use, directly attacks the lock-in problem of approach two. Some explicitly separate personal context, preferences, decisions, habits, from reference knowledge, your stable documents and notes, which is exactly the right architectural instinct.

The pros: portability across tools, real retrieval engineering instead of vendor defaults, and developer control for anyone building their own agents. The cons: another service in your trust boundary holding your most intimate data, another dependency that can change terms or die, and the same auditability challenge as built-in memory unless the service exposes its store honestly. For builders and power users, this layer is where the frontier is. For everyone else, it is early.

Approach four: the personal knowledge base as ground truth. The quiet winner among serious practitioners: maintain your knowledge in a system you already own, notes in Obsidian or similar, documents, saved research, and connect agents to it as a retrieval source, increasingly through MCP servers that expose your vault as governed tools. Your notes become your agent's semantic memory, and the curation you were already doing becomes context engineering.

The pros compound: the knowledge base outlives every tool, serves you directly as well as your agents, and keeps the human in the curation loop, which is the single best defense against rot and drift. The cons are honest work: retrieval over personal notes is only as good as their structure, and the approach rewards people who already keep good notes, which is not everyone.

The synthesis I practice and recommend: files for procedures and preferences, a knowledge base you own for reference knowledge, built-in memory as gravy with periodic audits, and memory services where you are building something custom. And one optimization principle over all of it: curation beats accumulation. The instinct is to give the agent everything. The results come from giving it the right small things, and pruning is a feature, not an admission of failure.

Optimization: The Techniques That Make Context Perform

Whichever approaches you choose, a shared toolkit of optimization techniques determines whether the assembled context actually performs, and they apply across the personal and enterprise divide with different emphasis. Five are worth knowing by name.

Compression and distillation. Raw history is a terrible context format: verbose, repetitive, and full of dead ends. The compression move distills it, summarizing long conversations into decision-and-fact records, collapsing repeated preferences into single statements, converting transcripts into structured notes. The memory services have industrialized this, with compression engines claiming token reductions up to 80 percent while preserving what matters, and the personal equivalent is a habit: end long working sessions by having the agent write the summary that future sessions will load, rather than dragging the whole transcript forward. Distilled context is not just cheaper. It performs better, because every surviving token earns its attention.

Hierarchical loading. Not all context deserves the window at once. The pattern that works is layers of increasing detail loaded on demand: a compact index or overview always present, telling the model what exists, with retrieval pulling full detail only when the task touches it. Personal knowledge bases do this naturally when structured well, an index note pointing to depth. Enterprise systems do it through metadata-first retrieval: the agent sees that a policy document exists, and fetches its content only when relevant. The anti-pattern is the flat dump, where the model wades through everything to find anything.

Relevance filtering and reranking. Retrieval's first pass is recall-oriented and noisy, so mature pipelines add a precision pass: reranking retrieved candidates against the actual query, dropping the marginally related, and enforcing a budget, the best five chunks, not the top fifty. The empirical finding that motivates the discipline: irrelevant context does not merely waste tokens, it actively degrades answers by diluting attention and inviting spurious connections. A smaller, sharper context beats a bigger, fuzzier one nearly every time, which remains the least intuitive and most valuable lesson in the field.

Freshness and expiry. Context should age on purpose. Facts carry dates, retrieval weights recency where recency matters, memories expire or demand revalidation after their shelf life, and superseded definitions are tombstoned rather than left to compete with their replacements. Personal version: date your notes and prune quarterly. Enterprise version: freshness metadata as a retrieval signal and staleness as an alert. Either way, the principle is the same: an unmanaged store drifts toward serving yesterday's truth with today's confidence.

Isolation and scoping. Finally, the technique that prevents the failure classes rather than optimizing the happy path: keeping context in scoped compartments, personal separate from shared, project separate from project, tenant separate from tenant, and assembling across compartments deliberately at runtime rather than pooling everything. Isolation is what makes the precedence rules of the next section enforceable, what keeps one project's assumptions from contaminating another, and, at enterprise scale, what makes multi-tenant agent products possible at all.

The meta-lesson across all five: context optimization is mostly subtraction. The novice instinct is additive, more history, more documents, more memory. The expert moves are distill, defer, filter, expire, and isolate. Every one of them removes something, and quality rises as they do.

Shared Context: A Different Game With Higher Stakes

Now scale the problem to an organization, and watch every assumption change.

The naive path is seductive: take what worked personally and multiply. Dump the company's documents into a vector store, wire retrieval into every agent, call it organizational memory. Every enterprise tried some version of this in 2024 and 2025, and the failures taught the industry what makes shared context a categorically different problem. Four differences do the damage.

Permissions are not optional. Personal context has one reader. Organizational context blends payroll discussions, customer contracts, engineering channels, and executive strategy, and a retrieval layer that ignores who is asking becomes a data breach with a chat interface. Shared context must be permission-aware at retrieval time: the same question from two employees should draw on two different slices of organizational knowledge, mirroring the access controls of the source systems, and keeping that mirror synchronized as permissions change is a hard, permanent engineering job.

Provenance is not optional either. In personal use, a slightly wrong memory costs you a correction. In an organization, an agent asserting "the discount policy is X" without a source is what one survey of the memory ecosystem aptly called a confident rumor. Enterprise context must carry citations: where a fact came from, when, and whether the source is still current, so humans can verify and systems can expire. Memory without provenance does not scale past one trusting user.

Consistency is a governance problem, not a retrieval problem. Personal context can tolerate my idiosyncrasies. Shared context has to answer questions like "what is our churn rate" identically for every asker, which means the organization's definitions themselves must be governed artifacts, owned, versioned, and machine-readable. This is exactly the semantic layer story I tell elsewhere, and the numbers make the case brutally: model accuracy on data questions rises from roughly 40 percent against raw schemas to over 83 percent when grounded in a governed semantic layer. Shared context without governed semantics is shared guessing.

And the content itself is a liability surface. Organizational knowledge bases are full of what knowledge managers call ROT: redundant, outdated, and trivial content, the 2019 policy that was superseded, the draft that lost, the wiki page nobody owns. Retrieval does not know the difference, so ROT becomes context poisoning at enterprise scale, and the companies getting results are the ones running deliberate knowledge hygiene, archival, ownership, freshness metadata, as a precondition for agent quality. Meanwhile, adversarial poisoning is real: any document an agent might retrieve is a prompt-injection vector, which is why the security guidance treats all retrieved content as untrusted input.

The stakes explain the investment. Around 75 percent of enterprises say they plan agentic AI deployments within two years, and the same industry surveys document deployments surging and then retreating as organizations hit exactly these walls. The pattern in the retreats is consistent: the model was fine, the context infrastructure was not.

The Enterprise Toolbox: Approaches to Governed Shared Context

Against those four challenges, the enterprise toolbox has matured into a recognizable stack, and the honest way to present it is as layers that compose rather than options that compete.

Governed retrieval. The evolved descendant of naive RAG: retrieval pipelines that enforce source permissions per query, attach provenance to every returned chunk, weight freshness, and log every retrieval for audit. The engineering burden concentrates in the unglamorous places, permission synchronization with dozens of source systems, index freshness, multi-tenant isolation, and this is where enterprise platforms and the context-infrastructure vendors earn their keep. The design principle to demand: the agent should never be able to retrieve what its principal could not read directly.

The semantic layer as shared meaning. For the structured half of organizational knowledge, the metrics, dimensions, and definitions the business runs on, the answer is the semantic layer, served to agents through MCP interfaces. I wrote a full companion article on this, so here I will compress: definitions authored once, owned and versioned, executable by any consumer, are what convert an agent from schema-guesser to endorsed-answer machine. In my house's version, Dremio's AI Semantic Layer pairs governed virtual datasets with AI-maintained wikis and labels, so the documentation agents ground on lives attached to the data rather than in a rotting wiki, and the whole surface fronts through an MCP server. Every serious platform now has its variant, and the emerging Apache Ossie standard aims to make the definitions themselves portable across all of them.

The catalog as the governance root. Underneath both retrieval and semantics sits the question of identity and access, and the open data world's answer generalizes beautifully to agents: catalogs like Apache Polaris give every principal, human or agent, role-based grants, and credential vending ensures an agent receives only short-lived, narrowly scoped access for each operation, with every touch logged against a real identity. This is the mechanical answer to the permissions challenge: agents governed exactly like employees, by the same infrastructure, auditable in the same trail. It is also, not coincidentally, where semantic definitions are increasingly governed as first-class assets, the convergence I have tracked across this whole series.

The context platform layer. Above the individual stores, a newer category assembles context across systems: platforms that ingest metadata and content from catalogs, BI tools, wikis, and existing semantic layers into one governed, AI-consumable surface, positioning as the organization's context hub. The instinct is right, business context is genuinely fragmented across tools, and the risk is equally clear: a context hub is a new center of gravity, and it inherits every synchronization and permission-mirroring challenge in one place. Evaluate these on their governance mechanics, not their demos.

And organizational memory, cautiously. The frontier question is whether organizations should have accumulating agent memory, lessons learned across all interactions, an emerging enterprise mind, and the research community is actively exploring exactly this. My honest 2026 counsel is caution: shared memory multiplies the poisoning, staleness, and provenance problems by every contributor, and the discipline that makes it safe, provenance-first writes, ownership, expiry, review, is exactly the discipline most organizations have not yet applied to their existing knowledge bases. Master governed retrieval and governed semantics first. They are the load-bearing walls.

The Seam: Where Personal and Shared Context Meet

The most interesting design problems live at the boundary, because every real workflow crosses it. Your work agent needs your personal preferences and your organization's truths in the same context window, and the two follow different rules.

The architecture that works is layered assembly with a precedence rule. Think of three tiers entering every context window: system context, the organization's non-negotiables, definitions, policies, access boundaries, drawn from the governed stack above. Shared operational context, the retrieved knowledge relevant to the task, permission-filtered and provenance-tagged. And personal context, your preferences, your style, your history with this work, drawn from your files and memory. The precedence rule resolves conflicts: for matters of fact and policy, shared governed context wins, your agent does not get a personal opinion about what churn means. For matters of style and workflow, personal context wins, how you like your summaries formatted is yours. Ambiguity between the tiers should surface to the human rather than being silently resolved, because silent resolution is how both drift and shadow-policy start.

Two boundary problems deserve explicit design rather than default behavior. Leakage upward: personal context must not flow into shared stores uninvited, your private notes on a colleague's proposal do not belong in the team's memory, which argues for explicit promotion, a human act that moves knowledge from personal to shared, with review, rather than automatic osmosis. And portability outward: your personal professional context, the preferences and practices that make agents useful to you, is increasingly a career asset, and the healthy pattern is keeping it in forms you own, files, your knowledge base, portable memory services, so that changing employers or tools does not lobotomize your workflow. Organizations, symmetrically, should be explicit that governed organizational context stays when people leave, which is one more argument for investing in the shared stack rather than letting institutional knowledge live in personal silos.

Handled well, the seam is where the magic actually happens: an agent that knows your organization's endorsed truth and your personal way of working is more useful than either alone, by a wide margin. Handled by default, the seam is where the incidents happen. Design it on purpose.

Two Worked Examples, One From Each World

Let me ground all of it in the two workflows I know best.

The personal case: my content operation. My weekly newsletters run on engineered personal context, and the architecture maps exactly to the approaches above. Procedural context lives in files: style rules, banned vocabulary, structural templates, sourcing requirements, versioned and loaded into every writing session. Reference context lives in sources I control: the Apache mailing list APIs I query for raw material, my own prior articles as consistency anchors. Session context is assembled fresh each week: this week's threads, this week's news, nothing stale carried forward by accident. Built-in memory features add convenience at the edges, and nothing I depend on lives only there. The results are the point: consistent voice across dozens of long pieces, no drift in the rules that matter, and total portability, because the context is files and feeds I own. The lesson I would generalize: the highest-return personal context is procedural, write down how you work, and the discipline that sustains it is pruning, my rule files get reviewed and trimmed, because every rule competes for attention with every other.

The enterprise case: the analyst agent. Now the organizational mirror. A financial services firm deploys an analyst agent for its business teams. System context: the agent authenticates as its own principal, the catalog vends scoped credentials per query, and the governed semantic layer supplies the endorsed definitions, revenue, exposure, churn, through the MCP surface, with the AI-maintained documentation attached. Shared operational context: retrieval over research, policies, and filings, permission-filtered so the agent surfaces to each banker only what that banker could read, every chunk cited with source and date. Personal context: each banker's preferences, coverage areas, formats, standing focus, layered in from their own tier, styling the answers without ever overriding a definition. When the agent reports a client's exposure, the number matches the risk system because both computed from the same governed definition, the citation trail satisfies the auditor, and the formatting matches how that banker works. Remove any tier and you get a recognizable failure: no personal tier, generic answers nobody adopts. No shared retrieval tier, confident rumors. No system tier, three exposure numbers and a compliance incident. The tiers are the architecture.

Anatomy of Three Context Failures, and What Fixed Them

Failure teaches faster than architecture diagrams, so let me dissect three composite failures drawn from patterns I encounter repeatedly, one personal, two organizational, each with the fix that actually worked.

The personal failure: the helpful assistant that slowly got worse. A writer's assistant, mine, in an early configuration, degrades over months. Suggestions get generic, old preferences resurface after being changed, and sessions start slow. The autopsy finds accumulation without curation: an instruction file grown to thousands of words of contradictory rules added incrementally, built-in memory holding stale facts alongside their corrections, and every session dragging maximal history. Nothing was wrong except everything being present. The fix was subtraction on schedule: the instruction file rewritten to a disciplined core, memory audited and pruned, session summaries replacing raw transcripts, and a monthly review made routine. Quality recovered within weeks, and the durable lesson entered my practice permanently: personal context has a maintenance schedule or it has a decay curve.

The organizational failure: the confident rumor at scale. An enterprise deploys a knowledge assistant over its document estate, and adoption starts strong until a business team makes a pricing decision on the agent's summary of a discount policy, which turns out to be the 2023 version, superseded twice. Trust craters, usage follows. The autopsy: retrieval was permission-aware and technically excellent, and the corpus was full of ROT, with no freshness signals and no provenance shown to users, so the system served outdated truth beautifully. The fix was less glamorous than the deployment: a knowledge hygiene program with owners and archival, freshness metadata as a first-class retrieval signal, citations with dates rendered in every answer, and a rule that uncited claims do not ship. Adoption recovered on the strength of a subtle change: users could now see why to trust an answer, which is the only durable form of trust an agent gets.

The governance failure: three numbers again, now automated. A company wires an agent to its warehouse through MCP, demo dazzles, rollout proceeds, and within a month the agent has produced three different quarterly revenue figures for three different askers, each computed from a defensible reading of the raw schema. The Monday meeting problem, faithfully automated. The autopsy is the shortest of the three: there was no semantic tier, so the agent guessed, and guessing at scale is fragmentation at scale. The fix is the one this article and its companion keep prescribing: the contested metrics defined and owned in a governed semantic layer, served through the same MCP surface, with the agent instructed to prefer defined metrics and to flag when a question falls outside them. The three numbers collapsed to one, and the flagging behavior turned out to be the sleeper win, a running inventory of which definitions the business still needed to write.

Three failures, one shape: in every case the model was fine, the context tier was missing or unmaintained, and the fix was context infrastructure plus context discipline. That is the field in miniature.

Decision Frameworks: Choosing Your Approach

Compressing everything into the guidance I actually give.

For individuals: start with files, today, whatever tools you use. Write the instruction file for your top recurring workflow, and you will feel the difference this week. Grow into a knowledge base you own as your reference layer, and connect it to your agents through MCP when your tools allow. Treat built-in memory as convenience and audit it occasionally. Adopt a memory service when you are building custom agents or genuinely working across many tools, and choose one that lets you export. Above all, curate: review your context artifacts monthly, prune ruthlessly, and remember that the goal is the right five pages in the window, not the complete ten thousand.

For organizations: sequence by load-bearing order. First, identity and access, agents as governed principals with scoped, short-lived credentials and full audit trails, because nothing else is safe without it. Second, governed semantics, the contested twenty metrics defined, owned, and served through an open interface, because that is where answer trust comes from. Third, governed retrieval with provenance over the knowledge that matters, paired with the knowledge hygiene program that makes it worth retrieving. Fourth, the personal tier, giving employees sanctioned ways to layer their own context so they do not build shadow systems. And only then, cautiously, shared memory. At every stage, prefer open interfaces, MCP for access, open catalog standards for governance, emerging open formats for semantics, because context infrastructure is a decade commitment and the portable choice is the safe one.

And for both: measure. Personal or enterprise, the health metrics are the same, how often answers required correction, how often stale context surfaced, how often the human bypassed the system, and the discipline is the same too. Context is a garden, not a warehouse. It rewards tending and punishes hoarding.

The 2026 Vendor Map, in One Honest Paragraph Per Category

Since evaluations always end with "so who do I look at," here is the map as it stands, with the usual caveat that this space reorganizes quarterly.

Personal memory and portability is the fastest-moving category: Mem0 for managed long-term memory and personalization, Supermemory for memory paired with retrieval over learned user context, Membase and its universal-memory peers for context that follows one person across many agent tools, and an open source field, Zep, Graphiti, Letta and company, iterating on the abstractions in public. Enterprise context infrastructure clusters around governed ingestion and retrieval platforms like Graphlit and its peers, offering permissioned, multimodal, audited context as a service, with the hyperscaler agent platforms bundling their own variants. The context hub category, Promethium's Context Hub, Atlan positioned as a context layer, Kaelio's open source ktx, aggregates semantics, lineage, and glossaries from existing tools into one AI-consumable surface rather than replacing them. The governed data side is my home turf and my earlier sections covered it: semantic layers from dbt, Cube, AtScale, the warehouse-native views, and Dremio's lakehouse-native approach, with Apache Polaris and Apache Ossie carrying the open governance and open semantics standards underneath. And the assistants themselves, Claude, ChatGPT, Copilot and their enterprise editions, keep absorbing the basics, projects, memory, instruction files, from below.

Read the map with the article's frame and the buying advice writes itself: individuals should own their artifacts and treat services as accelerators, enterprises should anchor on the governed layers and demand open interfaces from everything above them, and everyone should assume the category boundaries will keep dissolving, because context, memory, semantics, and governance are converging into one discipline before our eyes.

Questions I Hear Most Often

The recurring questions from both sides of my life, answered directly.

Don't giant context windows make all this engineering unnecessary? No, and the evidence has hardened rather than softened. Attention quality degrades as windows fill, costs scale with tokens, and irrelevant context actively harms answers, the rot phenomenon. Big windows changed the constraint from "what fits" to "what deserves to be there," which made curation more valuable, not less. The window is a stage, not a warehouse, and staging is the skill.

What is the difference between RAG and memory? Overlapping mechanics, different sources and lifecycles. RAG retrieves from reference knowledge, documents, data, things that exist independently of your interactions. Memory persists and retrieves from the interactions themselves, what was said, decided, and learned. A mature system runs both, and the design questions differ: RAG lives or dies on source governance and retrieval quality, memory on extraction accuracy, expiry, and provenance. The best mental model: RAG is the library, memory is the diary, and the context window is the desk where selected pages from each get laid out.

Should my company let agents remember things employees tell them? Only inside a governance design: memories attributed to identities, permission-scoped like everything else, provenance-tagged, expirable, and auditable. The failure mode is folk memory, unattributed claims accumulating into an unofficial policy layer nobody can inspect. If you cannot answer "who told the system that, and when," the system should not be remembering it.

How do we keep shared context from going stale? Treat freshness as metadata and ownership as mandatory. Every governed artifact, definition, document, memory, carries a source, a date, and an owner, retrieval weights recency, and staleness triggers review rather than silent serving. The organizations doing this well run knowledge hygiene as an ongoing program with the same seriousness as data quality, because for agents, it is the same thing.

Is context engineering a job now? Effectively yes, though the title varies. Inside data teams it looks like semantic layer and catalog ownership. Inside AI platform teams it looks like retrieval and memory infrastructure. Inside knowledge teams it looks like the hygiene and curation programs suddenly promoted from housekeeping to prerequisites. The common thread is treating context as a product with users and quality metrics, and the people who are good at it are becoming as valuable as the people who tune the models, arguably more, since context is where the failures actually live.

What should I do first, personally, this week? Write one file: the standing instructions for your most frequent AI-assisted task, your preferences, your constraints, your definition of good. Load it every session. Refine it for a month. That single artifact will teach you more about context engineering than any article, this one included, because you will watch quality respond to curation in real time.

Closing Thoughts

The split this article has walked, personal versus shared context, is really a split between two kinds of trust. Personal context earns trust through intimacy: the agent that knows how you work, kept honest by your own curation. Shared context earns trust through governance: the agent that knows what the organization means, kept honest by permissions, provenance, and owned definitions. The mistake of the past two years was treating them as one problem, scaling intimacy techniques to organizations and wondering where the audit trail went, or imposing governance weight on individuals and wondering why they routed around it.

The architecture that is winning treats them as layers with a seam designed on purpose: governed truth at the base, permission-aware knowledge in the middle, personal ways-of-working on top, assembled fresh into every context window with a precedence rule everyone understands. It is not a coincidence that the base layers are the open data infrastructure I write about every week, catalogs, semantic layers, and the standards making both portable. Context is where the AI era and the data era turn out to be the same era, and the organizations and individuals who treat their context as a curated, owned, governed asset are compounding an advantage that model upgrades cannot erase.

If you want to build those foundations properly, from the table formats and catalogs to the semantics and agent standards this all rests on, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O'Reilly, with further titles on lakehouse architecture, data engineering, and agentic analytics.

Browse the full collection of my books on data and AI at books.alexmerced.com.