Christian F. Rojas
ENES
All cases

Case 01

A documentation agent, chosen by measurement across three architectures

I wanted an agent that answers from our versioned technical manuals and always cites its source. I had three architectures built and evaluated, and chose with data, not opinion.

4.46 → 4.85
LLM-judge score (1–5), v1 RAG against v2 full context, on 26 golden questions
12/17 → 15/17
answerable questions with a grounded answer, v1 against v2
97%
Recall@10 for v3 on 34 paraphrased questions in Spanish and English
System diagram: three architectures for the documentation agent, each with its key figureThree architectures built in sequence. v1, RAG, uses a vector index and Claude via Bedrock: LLM-judge score 4.46 of 5 and 12 of 17 answerable questions grounded. v2 puts every manual of one version in the model’s context: 4.85 of 5 and 15 of 17. Both were scored on the same 26 golden questions. v3, highlighted, uses its own BM25 index and sub-agents with no vector database: Recall@10 of 97%, Recall@5 of 91% and MRR of 0.79 on 34 paraphrased questions.v1 · RAGvector indexClaude via Bedrockv2 · full contextall manuals of one versionin the model’s contextv3 · BM25 + sub-agentsno vector database3 model callssame 26 golden questions · LLM judge34 paraphrased questionsjudge 4.46 / 512/17 groundedjudge 4.85 / 515/17 groundedRecall@10 97%Recall@5 91% · MRR 0.79System diagram: three architectures for the documentation agent, each with its key figureThree architectures built in sequence. v1, RAG, uses a vector index and Claude via Bedrock: LLM-judge score 4.46 of 5 and 12 of 17 answerable questions grounded. v2 puts every manual of one version in the model’s context: 4.85 of 5 and 15 of 17. Both were scored on the same 26 golden questions. v3, highlighted, uses its own BM25 index and sub-agents with no vector database: Recall@10 of 97%, Recall@5 of 91% and MRR of 0.79 on 34 paraphrased questions.v1 · RAGvector indexClaude via Bedrockv2 · full contextall manuals of one versionin the model’s contextv3 · BM25 + sub-agentsno vector database3 model callsjudge 4.46 / 512/17 groundedjudge 4.85 / 515/17 groundedRecall@10 97%MRR 0.79
Fig. 01 · System diagram

My decisions 04

  • Choose the architecture with a measured evaluation: 26 golden questions in four categories, scored by an LLM judge.
  • Design v3 under the real constraint of having no vector database.
  • Rule out Pandoc for the Word-to-Markdown conversion, because content does not leave the team.
  • Use a fixed flow instead of a tool-using agent, because the agent node could not be forced to call its tool.

Assisted execution 03

  • Claude Code built the three versions and their n8n workflows under my rules.
  • Claude Code wrote the Word-to-Markdown converter with Python's standard library only, plus its tests: 29/29 unit and 9/9 flow simulation.
  • Claude Code ran the evaluation and computed the retrieval metrics.

Context

I lead technical documentation for a line of fraud-prevention products in enterprise cybersecurity. The manuals are long, they change with every version and they exist in Spanish and English. I wanted an agent that answers questions about them and always shows where the answer comes from.

What I decided

I decided not to pick an architecture by preference. Each one would be built and measured against the same questions before I chose.

The golden set had 26 questions in four categories: answerable, not covered by the documentation, about another product and about another version. Three of the four categories test whether the agent declines. A confident answer to a question the manual does not cover is the more dangerous failure.

For the third version I accepted a real constraint: no vector database was available. I also ruled out Pandoc for the document conversion. Our privacy policy does not allow it, and the rule is simple: content does not leave the team.

How it works

Three architectures, built in sequence:

  • v1, RAG. A vector index (Pinecone) and a Claude model through Bedrock, orchestrated in n8n.
  • v2, full context. Every manual of one version goes into the model’s context.
  • v3, no vector database. A BM25 index of its own, then three model calls: plan, select sections, answer. Answers carry verified [n] citations, and sub-agents talk through A2A-style contracts. The corpus is one product, 14 versions, 194 manuals and 16 release notes, in Spanish and English.

The v1 against v2 evaluation:

Metric v1 RAG v2 full context
Judge score (1–5) 4.46 4.85
Grounded answers (of 17 answerable) 12 15
Real grounding failures 2 0
Trick questions handled 9/9 9/9
Latency p50 10.7 s 13.8 s

v2 was more accurate and slower. That trade-off went to the decision-maker as a one-page summary.

What it produced

v3 reached a Recall@5 of 91%, a Recall@10 of 97% and an MRR of 0.79, on 34 paraphrased questions in both languages. Both recall figures beat their targets.

The conversion pipeline turned 194 Word manuals into Markdown with a minimum text retention of 99.4%, identical headings and zero alerts. It also found structural differences between the Spanish and English originals, and it corrected the document names across all 194 manuals.

What it does not prove yet

This is a proof of concept. In v3, one sub-agent is validated in n8n (7/7, about 1 s); the orchestrator is ready to import but not deployed. Nobody uses the agent in daily work yet, so I have no data on questions per day, satisfaction or the share of answers with a valid citation.

What I would do next

Add a grounding sub-agent and a feedback channel. Then put the agent in front of real users and measure questions per day, satisfaction and the share of answers with a valid citation.