Page-Aware RAG Architecture
Understand how UnieAI preserves page provenance while indexing and retrieving text, table, and figure chunks.
UnieAI Studio uses page-aware chunk RAG. It preserves each parsed page and its source metadata, then creates searchable text, table, and figure chunks that remain linked to their document and page.
Page-aware does not mean that every search result is a complete page. The primary search index contains chunks in Milvus; page records provide provenance, previews, document structure, and a fallback path.
Architecture at a glance
The PostgreSQL data and Milvus index serve different purposes:
| Storage | Main contents | Purpose |
|---|---|---|
| PostgreSQL | Documents, parsed pages, summaries, previews, PageIndex trees and nodes, processing state | Source of record, document structure, UI data, and page fallback |
| Object storage | Original files, page images, figure images, and other parsing assets | Visual previews and source assets |
| Milvus | Chunk content, Dense vectors, BM25 sparse vectors, and chunk metadata | Primary knowledge-base retrieval |
The current implementation does not use PostgreSQL pgvector for primary knowledge-base similarity search.
1. Parse according to the file format
Documents do not all pass through one page-by-page VLM OCR process.
| Source | Typical handling |
|---|---|
| PDF and images | Extract page text, layout, page images, and figures; use OCR or VLM where needed |
| Word and PowerPoint | Convert and retain document, heading, page, or slide structure |
| Plain text, Markdown, and source code | Decode text while retaining useful structural boundaries |
| CSV and Excel | Preserve sheets, columns, row groups, and table context |
Each normalized page can retain:
pageIndex- Raw or normalized text and a summary
- A preview and parser metadata
- Source kind and MIME type
- Page-image storage key and alternative text when available
A “page” is a source coordinate, not always a sheet of paper. In a PDF it normally maps to a physical page; in a spreadsheet or converted format it can represent a sheet, slide, or parser-defined unit.
2. Build document structure
During ingestion, Studio can build and store a PageIndex tree and flat PageIndex nodes. Nodes can describe a title, depth, page range, summary, and content. The root and top-level sections also contribute to document catalog summaries.
PageIndex currently supports document structure and internal retrieval surfaces. It is not the vector index used by the public knowledge-base Search page, /v1/kb/search, Remote MCP kb_search, or the standard Customize Model knowledge-base tool.
3. Create page-aware chunks
Parsed pages are transformed into chunks according to their content type. The system can use headings, paragraphs, semantic boundaries, line ranges, table rows, sheets, and recognized figures rather than one fixed-size window for every document.
A searchable chunk can retain:
| Field | Purpose |
|---|---|
documentId, chunkId | Identify the source document and exact indexed unit |
pageIndex, chunkIndex | Locate the source page and chunk order |
documentTitle, sectionTitle | Preserve human-readable document structure |
pageImageKey, figureImageKey | Link the chunk to visual assets |
sourceKind, mimeType | Describe the original format |
content, embeddingInput | Store displayed text and the contextualized text used for embedding |
Page labels, section names, and document titles can be included in the embedding input without rewriting the displayed source text. Contextual Retrieval may add another document-locating prefix when configured.
Most page-oriented strategies keep chunks within a page, but the architecture does not promise that every chunk equals exactly one page or that every strategy can never span pages. pageIndex is the chunk's source anchor.
4. Index chunks in Milvus
Each chunk is indexed with:
- A Dense embedding for semantic similarity.
- A BM25 sparse representation for lexical matches.
- Document, page, section, source-format, and image metadata.
The configured embedding dimension must match the active Milvus collection. Changing the embedding model or dimension requires the platform's reindex or migration workflow; existing vectors do not automatically become compatible.
5. Retrieve with Hybrid Search
Knowledge-base searches run Dense and BM25 searches, merge the candidates, calculate a Hybrid relevance score, apply Min score, and return up to the requested Limit or model Top K.
Results are chunks and can include:
- Content and Hybrid score
- Knowledge base, document, and chunk identifiers
pageIndex,chunkIndex, and section title- Page- or figure-image URLs when assets exist
- Retrieval warnings and other metadata
The API-level pageIndex is zero-based. Studio normally displays it as a one-based page number, so pageIndex: 0 is shown as page 1.
Page and figure URLs can be temporary signed URLs. Use them for the current response or preview; do not treat them as permanent public asset URLs.
6. Merge across linked knowledge bases
When a Customize Model links multiple knowledge bases, Runtime searches the relevant sources, merges and deduplicates chunks, sorts them by score, and applies the final Top K after the cross-knowledge-base merge.
Top K is therefore a global result cap for the model request, not a separate allowance for each knowledge base. A single document can contribute more than one chunk when several passages are relevant.
7. Page fallback
Customize Model retrieval can fall back to stored page content when chunk search fails, or when no chunks are found and the current threshold behavior permits fallback. These rows are marked with retrievalSource: "fallback_pages".
Page fallback:
- Keeps the conversation usable when the primary chunk index is temporarily unavailable.
- Can return page text or summaries with page metadata.
- Is not another selectable search mode.
- Does not produce a Milvus Hybrid score and should not be evaluated as if it did.
An explicitly supplied Min score can prevent an empty-result fallback from bypassing the intended threshold. A genuine search failure can still attempt the resilience path.
Customize Model behavior
Configure the model in these locations:
- Data → Knowledge Base: choose which knowledge bases the model can search.
- Behavior → Agent Loop: configure Top K, Min score, Agentic RAG, and its maximum iteration or step limits.
With Agentic RAG disabled, Runtime can retrieve chunks before the base-model call and inject the selected context. With Agentic RAG or Agent Core enabled, the model can decide when to call kb_search, refine a query, and search again within its execution limits.
There is no fixed rule that rewrites every empty query exactly eight times. Iterative behavior depends on the selected Agent Core and the model's saved execution settings.
The public OpenAI-compatible request should not rely on knowledgeBaseIds or max_retrievals to replace a Customize Model's saved knowledge bases and Top K per request. Configure those values on the Customize Model.
Citation and answer quality
Page provenance improves traceability, but it does not guarantee that the model will cite the correct page automatically.
- Ask the model to cite the document and page when those fields are present.
- Verify that the retrieved chunk actually contains the claim.
- Remember that
pageIndexcan be a parser unit for non-PDF formats. - Treat a chunk anchored to one page as source metadata, not proof that every character originated from only that page.
- For important answers, let users open the page or figure preview and compare it with the original file.