Choosing a RAG Tool: Matching Document Type to Error Risk

First: Do you need to search or generate an answer?

Before comparing tools, define what the system must do. Keyword search finds literal matches; semantic search attempts to retrieve passages related by meaning; a Retrieval-Augmented Generation (RAG) system adds retrieved results to the context of a language model so it can draft an answer. These are distinct capabilities, even if a product combines them.

A conventional search engine may suffice if users know the specific term, title, or identifier to look for and need to open the original document. Semantic search is useful when the query and the document use different words to express a similar idea. RAG makes sense when you need to synthesize information from multiple passages into a conversational answer, with mechanisms to consult the sources. It does not make retrieved information automatically true, nor does it turn a well-written response into evidence.

Criterion 1: Document format determines preparation

File format is not an ingestion detail: it conditions what text and structure reach the search. A plain text document is usually easier to index than a scanned PDF, a sheet with merged cells, or a report where titles, notes, and tables depend on each other. For page images, OCR may be required. For tables, columns, and headers, extraction that preserves relationships between data is necessary.

Chunking—the division of documents into retrievable fragments—must also respect the content. If a table is cut between rows or a figure is separated from its header, the fragment may appear relevant yet remain ambiguous. In long texts, it is advisable to test limits based on sections or semantic units, preserve metadata, and review real examples of indexed fragments. There is no universal optimal size: it depends on the type of query, the content, and the model receiving the context.

Criterion 2: Currency, permissions, and traceability

If documents change frequently, index updating is part of the design. Ingestion needs a policy to detect additions and changes, reflect current versions, and remove content that should no longer appear in results. Measure how long it takes for a change to be available via queries: a promise of updating is not the same as a test in your data flow.

For a sensitive query, the tool must maintain a verifiable link between each answer and its sources: document identifier, version or date, section or page, and fragment used, as allowed by the corpus. It must also allow reconstructing what was retrieved in a query and applying permissions before content reaches the model. A visible citation can help review, but it does not by itself prove that the claim is supported or that the source is the appropriate authority.

The EU AI Act does not turn every RAG system into a high-risk system nor establish a single citation specification for any tool. Classification depends on use and applicable conditions. If the system is part of a regulated use case or influences sensitive decisions, relevant obligations must be identified with specialized advice, and records, supervision, and access control must be designed accordingly. The sources available here are not sufficient to detail all legal requirements by sector or system.

Criterion 3: Retrieval quality, latency, and cost

There is no single metric that validates a complete RAG system. In retrieval, recall measures if the expected relevant passages were found; precision measures what proportion of retrieved results is pertinent. MRR values the position of the first relevant result; NDCG considers the quality and order of multiple results. These are useful metrics for comparing retrievers on a set of queries with expected answers or passages, but they do not prove that the generated answer is true, complete, or legally correct.

Evaluate separately the fidelity of the answer to the context, the coverage of necessary information, and the correctness of citations. Define examples where the correct answer is to abstain, including cases with insufficient documents, contradictory sources, or insufficient permissions. Automatic metrics can help review many executions, but it is advisable to cross-check them with human judgments on representative queries, especially when errors have significant consequences.

Measure end-to-end latency and break it down: search, re-ranking—reordering candidates by utility—model calls, and other stages. More retrieval steps can increase time and cost; they should not be added without checking that they improve quality for the real problem. Cost per query also depends on context length, number of calls, volume processed, and infrastructure. Available sources do not set universal limits on document size or table complexity for commercial tools: those limits depend on the product, configuration, and date.

Three scenarios and which tool to test first

Unstructured documents: prioritize the ability to extract content before comparing models. Test OCR when there are scanned pages, parsing that preserves tables, and chunking that respects sections. If queries contain names, codes, or exact references in addition to concepts, compare lexical and semantic search, including a hybrid combination. Re-ranking can be useful if the initial set contains pertinent passages but orders them poorly; it does not repair an extraction that lost information.

Frequent updates: define freshness objectives and verify the complete cycle of updating, replacing, and deleting. Keep dates and versions as retrievable metadata. If an answer depends on information from more than one source or requires decomposing the query, an iterative strategy can help, but it adds steps. Start with standard retrieval if a search in one index suffices; then compare the improvement with the increase in latency and cost.

Legal or scientific traceability: require locatable sources, records useful for later review, permission controls, and abstention criteria. Generation does not replace specialist interpretation nor validate the authority or validity of a document. For high-impact claims, review citations and original context; record corrections and evaluate if the system should limit itself to locating evidence rather than drafting conclusions.

Limits: Relevance does not mean truth

A retrieval system orders candidates based on matching or relevance signals; it does not certify that a document is correct, official, current, or applicable. An answer can cite a pertinent fragment and still generalize it, omit a condition, mix versions, or misinterpret a table. It may also fail to find the decisive source and produce a fluid answer based on incomplete information. Therefore, the presence of a citation is not equivalent to factual verification.

Defenses must correspond to the type of risk: filter by permissions during retrieval, preserve provenance and version, cross-check claims with passages, detect conflicts, test questions without sufficient answer, and allow abstention. In high-impact contexts, incorporate human review and defined responsibilities. Supervision must include samples of queries and real failures after deployment, not just an initial evaluation.

RAG also does not decide which sources deserve authority nor resolve by itself contradictions between documents. Those decisions require governance rules—for example, priority by status, date, or owner—and, in sensitive cases, human judgment. If a document search without generation satisfies the need, it can be a simpler and easier-to-audit option. The right tool is the one that solves the task within measured limits, not the one that incorporates more components.

Practical criteria for deciding

Start by describing real queries and the consequence of answering wrong. If the task is to find documents, test keyword or semantic search first. If synthesizing passages is needed, compare RAG with a search baseline and evaluate retrieval and generation separately. If documents are hard for machines to read, solve OCR or parsing first; a generative model does not reliably reconstruct what never reached its context.

To compare providers, request tests with your own corpus or a representative sample, and verify specific limits in the current documentation of the product to be contracted. Check supported formats, maximum sizes, table handling, updates, permissions, record export, and costs under a planned query load. Do not extrapolate a figure from one configuration or provider to all systems.

The final decision can be gradual: search engine first, hybrid search if exact and semantic terms coexist, re-ranking if it improves result order, and RAG if synthesis adds measurable value. Add iterative retrieval only when queries truly require multiple steps. Before deploying, establish approval criteria, data owners, review rules, and a procedure to withdraw or correct problematic answers.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.