Figure Extraction

Learn how the VLM detects, crops, and describes visual elements and how those assets enter the knowledge base index.

Figure Extraction is part of the VLM document-parsing pipeline. The platform identifies charts, diagrams, photos, and other visual elements on PDF pages or images, stores cropped assets, and adds searchable text descriptions to the knowledge base index.

Model prerequisites

An administrator must first configure an available VLM parser model under Admin panel → System Models → UnieAI VLM parser. The platform runtime must be able to call the model, and the uploading user must have access to it for parsing and billing to complete.

The standard knowledge base upload flow also requires a configured KB Chunking Task Model. When a required model is missing, file upload may be unavailable or the parsing job may fail and display an error in the document list.

Figure detection runs automatically when a PDF or image enters the UnieAI VLM parser pipeline. Whether each image also receives a standalone chunk is controlled by Settings → Parsing → Create an independent Chunk for each Figure on the knowledge base and is off by default.

Supported sources

PDF: Each page is rendered as an image before text parsing and figure detection.
Image files: The full image goes directly through VLM parsing. If no individual region is detected, the platform attempts to treat the whole image as a photo and caption it.
PowerPoint: Supported presentation files are converted to PDF first and then use the PDF pipeline.

Word, plain-text, Markdown, and spreadsheet files use other parser paths and are not guaranteed to produce the same figure artifacts.

Processing flow

1. Detect visual elements

The platform asks the VLM for a JSON array in which each result contains:

type: chart, diagram, figure, photo, logo, icon, table_graphic, or illustration
bbox: [left, top, right, bottom]
description: a short detection-stage description

The bounding box uses normalized coordinates from 0 to 1000, with [0, 0] at the top left and [1000, 1000] at the bottom right. Values are clamped to the valid range, and malformed results do not enter the crop step.

2. Crop and generate a detailed caption

The platform converts normalized coordinates to pixels, clamps them to the page boundary, and skips regions narrower or shorter than 20 pixels. Valid regions are stored as PNG assets.

Each successfully cropped figure is then sent to the VLM again to attempt to add:

A detailed description
Verbatim text visible inside the figure
Language
The closest page heading
A nearby paragraph

If captioning fails, the platform retains detection-stage data and continues processing the other figures instead of failing the whole document because of one image.

How figures enter chunks

Default: keep figures in page text chunks

PDF parser output can contain figure_N placeholders. The platform pairs them with uploaded crops by order and type, then writes a stable image asset key into Markdown image syntax. Uploaded figures with descriptions that were not referenced by the page text are appended to that page's text.

A single text chunk can therefore carry multiple images and retrieve them with their surrounding text. By default, the platform does not create standalone Figure chunks for those images.

Optional: create standalone Figure chunks

To retrieve a specific image directly, open the knowledge base's Settings → Parsing, enable Create an independent Chunk for each Figure, then upload a new document or re-index an existing one. When enabled, every Figure with a cropped image also receives a standalone chunk. Its searchable text combines the figure type, page, description, visible text, nearby heading, and nearby paragraph.

{
  "sectionTitle": "Figure 1",
  "figureImageKey": "kb123/doc456/figures/page-0-fig-1.png",
  "pageImageKey": "kb123/doc456/pages/0.png",
  "metadata": {
    "chunkingStrategy": "figure",
    "figureType": "chart",
    "figureDescription": "A bar chart showing revenue by quarter",
    "figureLanguage": "en",
    "figureVisibleText": "Q1 Q2 Q3 Q4",
    "figureAnchorHeading": "Quarterly revenue",
    "figureNearbyParagraph": "Revenue increased in each quarter."
  }
}

Figure chunks use descriptions and visible text for Dense embedding and BM25 indexing. They already contain visual context, so Contextual Retrieval skips them.

Image assets and URLs

The database and vector index store stable asset keys, not expiring signed URLs. When the document chunks page, search API, or RAG response needs an image, the platform uses the knowledge base asset API to generate a usable URL at response time.

This prevents stored chunks from breaking when a signature expires and keeps long, search-irrelevant signatures out of embeddings.

View and delete figures

On a knowledge base document's chunks page:

The page view displays detected bounding boxes over the original page image.
The chunks view can preview inline images in text chunks and standalone Figure chunks.
A user with edit permission can delete a standalone Figure chunk.

Deleting a Figure chunk also removes its image asset, scrubs it from saved parser artifacts, rebuilds the document chunks, and updates the knowledge base snapshot and audit history. The deleted figure therefore does not return after a refresh or re-chunk operation.

Best-effort behavior and limitations

A VLM can miss figures, select an incorrect region, or classify decoration as a meaningful visual.
A result without a valid bounding box, a crop that is too small, or a failed asset upload may not produce a previewable crop.
PDF placeholders are first paired by order and type. When no reliable match exists, the original placeholder may remain or an unused detected figure may be used as fallback.
When no region is found in a plain image, the platform attempts to caption the full image. This fallback can still produce no result if the VLM call fails.
Figure search operates on VLM-generated text, not direct image-pixel similarity.

Cost, latency, and data privacy

Each page requires VLM parsing and figure detection. Every successfully cropped figure may add another caption call, so image-heavy files take longer and consume more model usage.
Figure captions are processed concurrently, but actual concurrency depends on deployment settings and the number of figures.
Page images, cropped images, and nearby text are sent to the administrator-configured VLM endpoint. Before using an external model service, review data residency, retention, confidentiality, and personal-data requirements.
Captions are model-generated. Manually inspect figures when exact numbers, legends, axes, or domain terminology matter.