Agent Team in Practice (6): Knowledge — Em's Wiki and 'Not Everything Belongs in Context'

In one line: An Agent Team’s knowledge shouldn’t be scattered across each Agent’s Context — it needs a centralized, compilable, searchable knowledge base.

One evening I told Em to ingest 10 conversation logs — records of my interactions with different Agents throughout the process of building the entire Agent Team, including how I coordinated Em and C7 for parallel processing, how I discussed GM’s design with Agent G, and how I manually handed off one Agent’s output to another, among other records. After the ingest completed, Em’s wiki gained 3 brand-new concepts (bootstrap-agent-pattern, prompt-relay-pattern, cross-machine-registry-pattern), updated 3 entity pages and 2 existing concepts, and produced a source summary containing a complete timeline of all 10 conversations, 5 human coordination patterns, 10 architectural decisions, and “7 patterns GM needs to learn.”

At that moment, I finally felt like an Agent had “remembered” the things we’d done together. That sudden feeling didn’t come from a larger Context Window (around that time Opus models’ Context Windows grew from 200K to 1M tokens) — it came from the knowledge we’d saved starting to get structured. Discussions, decisions, and Patterns that had been scattered across different session conversation histories, after Em’s ingest, became wiki pages that any Agent could query at any point in time — and the pages cross-referenced each other! That meant Em didn’t just store knowledge and structure it — it also built the connections between pieces of knowledge. That brought a new dimension to Context.

Step Back and Ask: Why Do We Need a Knowledge Agent?

In earlier articles I mentioned that building the Agent Team consumed roughly 800 pages’ worth of output tokens, and nearly all of that output was documents scattered across different conversation logs, different design files, and different Agents’ working directories. Without someone systematically organizing it all, that knowledge was just text sitting somewhere — and unless you remembered to tell an Agent to go read it, that text wasn’t usable knowledge. Agent Em, designed specifically to solve that problem, was the answer I needed. For now, Em doesn’t do anything else — it only does three things: ingest (converting raw materials into structured wiki pages), query (finding relevant knowledge in the wiki to answer questions), and lint (checking the wiki for consistency and completeness — more on how Em grew later). It serves as the entire Agent Team’s knowledge compiler [2][7] — raw data in, structured (compiled) knowledge out.

In the Context Engineering article I mentioned the principle of JIT loading [1]: rather than stuffing everything into Context and waiting for it to be needed, you load what you need when you need it [5]. Em’s wiki is the concrete embodiment of that principle. Agents don’t need to remember all knowledge — they just need to know to “go ask Em,” and Em will find the relevant pages, knowledge, and indexes from the wiki and provide them. (Later this evolved so Agents can query on their own.)

Of course, there are two different paths here. For many people, all they need is to improve their existing workflow after an ingest — in other words, let knowledge drive the Agent Team to evolve. The loop of knowledge driving a single Agent forward might be enough, without needing a dedicated knowledge system at all. I agree with the first part of that view, but it’s not quite what I wanted to do. Because of how Agents tend to behave (fixing what’s right in front of them, preferring to add over remove, self-validating their own errors), having an elevated, independent layer of knowledge for auditing, reviewing, reusing, evolving, and eventually backtesting or integration is necessary. So I chose the other path: build the knowledge first, then let knowledge drive improvement and evolution. It’s the same as the three-layer methodology, Playbook, and Plan I mentioned earlier — if we focus on building and updating Playbooks and Plans but forget to update the methodology, we’re stuck starting over whenever we hit a scenario that doesn’t map onto what we have. But if we preserve the methodology, we have more flexibility and fallback options in exactly those situations. That’s also the other important responsibility I want Agent Em to carry.

raw → wiki → inbox: Three Stages of Knowledge

Em’s knowledge Pipeline has three stages, each handling a different kind of data.

raw/: Where raw materials live. Em doesn’t do any processing on the Raw data — this might be a conversation log, an external document, the contents of a GitHub repo, or even a scraped DeepWiki page. Em never modifies these materials, it only “extracts” knowledge from them.

wiki/: Em’s core output — essentially a structured knowledge base, but with four types of pages: concepts (ideas like Context Engineering or the three-layer methodology framework), entities (people, things, tools — like each Agent, external tools, or the author), sources (source summaries documenting the raw materials and outputs from each ingest), and analysis (cross-concept synthesis, like methodology recommendations for GM). Every page follows a fixed structure (title, type, date, source tags), making it easy for Agents to read and understand [8], and pages are connected via cross-references so you can find related material when you need it.

inbox/: Pending messages from other Agents. When C7 completes a skill entry, or Dm finishes building a new Agent, they can drop a notification into Em’s inbox telling it “there’s something new worth ingesting into the wiki.” This mechanism is still mostly manual for now (usually I’m the one telling Em “go ingest this”), but I want the design to eventually have this automated and coordinated by GM.

It looks mechanical, but steps 3 and 4 are where the most judgment is required. Em has to decide: is this piece of knowledge a brand-new concept, or does it supplement an existing one? If it’s a supplement, which page and which section gets updated? If multiple concepts are involved, how should the cross-references be structured? These judgments determine the quality of the wiki, and the quality of the wiki determines whether other Agents get useful results when they query it. This is adapted from a methodology I found online, and it’s been working well so far.

Three Ingest Stories

Describing the flow abstractly only gets you so far. Let me open it up with three real ingest cases.

Case 1: Ingesting an External Plugin Repo

The first was ingesting the knowledge from my own Claude Code Plugin repo (emilwu-plugins), which contains three plugins already in production use: investigate (parallel cross-repo investigation), claude-insights-command (session analysis reports), and claude-insights-skill (same thing but in Skill form).

The instruction I gave Em wasn’t just “ingest this repo” — I attached a detailed extraction focus, telling it “there are three patterns worth extracting here”: the Parallel Investigation Pattern (Explore → Decide → Execute three phases), the Session Analysis Pipeline (a five-stage data pipeline with two-layer parallel subagent processing), and the Plugin Distribution Pattern (the packaging structure of hooks + scripts + prompts). This gave the ingest an initial framework to work from.

After Em read 16 files, it produced 3 new pages (a source, an entity, and a new concept: multi-phase-llm-pipeline) and updated 4 existing pages (subagent-pattern gained two real-world cases, plugin-packaging got distribution details filled in).

The key design decision here was giving Em the extraction direction inside the ingest prompt itself, rather than letting it judge on its own what was worth extracting. Because I judged that Em’s Context didn’t include “how long this Plugin has been running in production” or “which patterns have been reused by other projects” — without that direction, it might focus on the wrong things. This is a similar judgment to what I discussed in the calibration article about “what to spell out vs. what to let go”: spell out the goals and constraints, let go of the execution path.

Case 2: Ingesting 10 Conversation Logs from When the Agent Team Was Being Built

The second case is the one I described at the start — ingesting 10 conversation logs from my interactions with different Agents while building the Agent Team. This ingest was a little different: I gave Em the instruction to start reading files following the standard flow, then immediately interrupted and gave a richer Prompt instead.

Why? Because I realized the standard flow wasn’t enough. These conversation logs weren’t “knowledge documents” — they were “records of work in progress.” Em needed to extract patterns from the work process, not just pull out the highlights of the conversation content. Those two approaches produce very different results. If you’ve ever had an AI summarize your conversation logs for improvement, you’ve probably noticed that giving direction vs. not giving direction makes a huge difference. So in the second Prompt I wrote: “This is the process of how I built the entire Agent Team. Extract the knowledge embedded in the process — focus on how I coordinated these Agents to get things done, because that’s what Agent GM needs to learn. The communication patterns in these conversations are also important, because they serve as the basis for designing the inbox and communication mechanisms.”

What made that Prompt matter: ingest quality depends not just on Em’s ability, but also on the direction and guidance the human provides. If you just hand Em a pile of files and say “ingest,” it will use its own judgment to decide what’s important. For a GitHub repo, an article, or a news piece, that might be fine — but for conversation logs that already have context, Agent doesn’t have the context that’s in your head, like “the coordination patterns in these conversations are things GM will need to learn.” Without that, the extraction can go off in the wrong direction. That said, we also have to be careful not to over-specify — you don’t want the Agent trying to piece together something from the conversation that simply isn’t there.

In the end, Em extracted from 10 conversations: bootstrap-agent-pattern (the pattern of Agent G as a temporary builder), prompt-relay-pattern (the current state of humans manually passing prompts between Agents — exactly what GM needs to automate), cross-machine-registry-pattern (Agent registration and sync across machines), and a source summary organizing “5 patterns of Human as Coordinator.” All of it laid the foundation for the Agent Team’s continued development.

Case 3: cross-concept synthesis

The third case was a hybrid application — we had Em do an ingest and a cross-concept synthesis at the same time.

During GM’s design phase, Dm’s Architect needed a GM Methodology recommendation to work from. But GM’s Methodology couldn’t be derived from a single source — it required cross-referencing multiple concepts already in the wiki: agent-team-architecture explained GM’s position in the architecture, context-flow-decision-model explained when GM should use a Skill vs. a Subagent vs. Agent Teams, methodology-playbook-plan described what a Methodology itself should look like, completion-reporting-protocol explained how Agents report back after completing work, and prompt-relay-pattern cataloged what humans are currently doing manually. So I had Em read 12 concept pages, 5 entity pages, and 2 source summaries, then produce a synthesis analysis of GM’s methodology. It contained 7 methodology principles (each traceable back to a specific concept in the wiki), scenario pattern recommendations for GM (patterns distilled from 16 use cases), and Context design recommendations (what belongs in CLAUDE.md, which knowledge should only be loaded on demand). With all of that in place, Dm could begin designing GM.

This is Em’s most valuable capability: not just storing knowledge, but connecting knowledge scattered across different pages to produce insights that no single page can provide. And this is fundamentally different from having an Agent ingest a session’s Context every time you finish a conversation, or doing a weekly review of all your findings — because it’s building connections between knowledge and frameworks in a structured, intentional, directed way. That’s architecturally different from plain summarization, refinement, or distillation. After Dm’s Architect read Em’s synthesis, combined with the 16 scenarios in the design inputs, it designed GM’s development plan without needing to come back and ask Em “what are the specs for that protocol?” — because the synthesis had already integrated all the relevant knowledge together. And if you’re just doing real-time improvements from each work session or weekly reports, you won’t arrive at this kind of insight or be able to use it to build something larger and more complete.

What Happens When the Wiki Gets Big?

At this point, everything about Agent Em’s wiki sounds great: knowledge is structured, cross-referenced, capable of cross-concept synthesis, queryable by other Agents at any time. But just as we get here, another problem was about to surface — performance slowed down.

Because as the wiki grew from 20-something pages to 50-something pages and then to 120+ files, every /query requires Em to first read index.md to find the relevant pages, then read those pages, synthesize, and answer the question. As the wiki keeps growing [4], index.md itself gets very long, and Em — in order not to miss anything (I don’t allow it to skim) — often reads more pages than actually needed, because it can’t tell from the title alone whether a given page is relevant to the question.

This is a linear token cost scaling problem [3][10] — the larger the wiki, the more tokens get read per query, and most of what gets read ends up unused. This is actually the flip side of the Context Stuffing problem discussed in the Context Engineering article: you’re not loading everything into Context at startup (that’s eager loading), you’re being forced to read too much at query time (that’s a search problem).

More specifically, the issue is that Em’s current query workflow is missing a search layer. The current flow is: read index → read pages → synthesize answer. The truly ideal flow should be: use semantic search to find the most relevant pages → read only those pages → synthesize answer. The difference is whether the first step is “traversal” or “search.” Traversal cost grows proportionally with wiki size; search cost is (in theory) constant.

So I introduced QMD (a semantic query tool based on vector search [6]), letting Em’s queries first use semantic search to narrow the scope before reading the relevant pages. But the QMD integration story is more complex — it involves a three-phase design — so I’ll save that for the next article on the memory system.

Knowledge Is More Than Storage

Coming back to Em: during the build process, one small moment stuck with me.

After the CLAUDE.md refactor mentioned in the previous article (dropping from 99 lines to 47), I had Em update its own wiki entity page to document the refactor. That was a small session that lasted only 4 minutes. Em updated wiki/entities/agent-em.md, added a description of the wiki-schema skill in the Configuration section, then recorded in wiki/log.md the rationale for “extracting Page Format / Workflows / Conventions into a skill.”

An Agent recording its own architectural changes in its own knowledge base — it sounds very meta, but it demonstrates something important: the wiki isn’t just “a database for other Agents to query.” It’s also every Agent’s own growth record. The next time someone asks “when did Em do a refactor and why,” the answer is right there in Em’s own wiki, without needing to dig through conversation logs or release notes.

This surfaces another observation. In traditional software development, knowledge management is usually an afterthought: write the code first, then document it (if there’s time). But in an Agent Team, knowledge management is a core part of the workflow — because an Agent’s capability is directly determined by its Context. The more complete the Context, the more precise the Agent’s output. If a concept isn’t in Em’s wiki, other Agents won’t know it exists and won’t factor it into their design decisions [11] — and the actual output could end up missing things. So calling Em the brain and the source of the entire Agent Team’s development isn’t really an overstatement.

800 pages’ worth of output tokens — the value of that output isn’t in “how much was written,” but in “how much was structured.” Scattered conversation logs are raw data. Wiki pages that have been ingested by Em are knowledge. And knowledge’s value lies in the fact that it can be found, cited, cross-linked, and used as a basis for decisions by other Agents when they need it.

The Complete Knowledge Pipeline

If you were to draw the full knowledge Pipeline, it would look something like this:

Every step in this pipeline is compression and structuring of Context: from raw data full of noise and repetition, to concepts filtered and compressed by Em, then cross-references between concept pages that turn knowledge into a network, and finally the QMD index that means queries don’t need to traverse the entire wiki — each stage hands the baton to the next, ultimately producing a system where other Agents can load the knowledge they need only when they need it.

This is the practical implementation of Context Engineering’s JIT loading principle at the knowledge management level [9]: knowledge shouldn’t exist in every Agent’s Context — that wastes Agent Context and Token. Knowledge should live in a centralized, compilable, searchable knowledge base, where Agents query it when they need it, and once they have what they need they release that portion of Context — making room for the next step of work. That’s what an efficient system looks like.

So, the most underestimated role in an Agent Team is actually Em — because nobody thinks past document management matters, until the day they need to find the reason behind a decision that was made back then and can’t find it anywhere.

Maybe that also explains why Em was the first Agent to be built: because before any of the other Agents could start doing their work, there needed to be somewhere to store knowledge — and let the system know “what we already know.”

References

[1] Anthropic — Effective Context Engineering for AI Agents (official guide: selective loading over context stuffing) https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

[2] The New Stack — 6 Agentic Knowledge Base Patterns Emerging in the Wild https://thenewstack.io/agentic-knowledge-base-patterns/

[3] arXiv — Solving Context Window Overflow in AI Agents https://arxiv.org/html/2511.22729v1

[4] AI Pace — Context Engineering: Mitigating Context Rot in AI Systems (“context rot”: the larger the context, the lower model reliability) https://medium.com/ai-pace/context-engineering-mitigating-context-rot-in-ai-systems-21eb2c43dd18

[5] Jentic — Just-In-Time-Tooling: Scalable, Capable and Reliable Agents (JIT architecture: load on demand rather than preload) https://jentic.com/blog/just-in-time-tooling

[6] arXiv — Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (counterpoint: industry leans toward RAG + embedding over structured wiki) https://arxiv.org/abs/2501.09136

[7] InfoWorld — Anatomy of an AI Agent Knowledge Base https://www.infoworld.com/article/4091400/anatomy-of-an-ai-agent-knowledge-base.html

[8] arXiv — MAGMA: A Multi-Graph based Agentic Memory Architecture (counterpoint: graph structure may be more effective than flat wiki) https://arxiv.org/pdf/2601.03236

[9] Context Studios — From Mode Collapse to Context Engineering (“focused 300-token context outperforms unfocused 113K-token context”) https://www.contextstudios.ai/blog/from-mode-collapse-to-context-engineering-how-we-build-reliable-ai-systems-2026

[10] Factory.ai — The Context Window Problem: Scaling Agents Beyond Token Limits https://factory.ai/news/context-window-problem

[11] Meta Engineering — How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines https://engineering.fb.com/2026/04/06/developer-tools/how-meta-used-ai-to-map-tribal-knowledge-in-large-scale-data-pipelines/