Data Foundry De-identification
Run de-identification jobs on knowledge-base chunks, monitor processing, and manage the resulting versions.
Data Foundry detects selected types of personal or sensitive information in knowledge-base chunks and replaces the original text with fixed placeholders. It operates after documents have been parsed and indexed; it is not a pre-upload file converter or download tool.
A successful Data Foundry job immediately updates the currently searchable chunk and its vector index. Confirm the selected scope and data types before starting. You can restore an earlier version from the chunk history if needed.
Before you begin
Confirm that:
- The platform license includes the Knowledge Base and Data Hub features.
- An administrator configured the Data Foundry base URL, task model, temperature, and pricing.
- The target knowledge base contains parsed and indexed chunks.
- You can access the target knowledge base and understand that the job changes its retrieval content directly.
- The personal or organization wallet has enough balance for model-token usage.
When you first open the task page, Studio displays a model and pricing confirmation if the administrator's Data Foundry task model is not yet in My Models. Select Add to My Models before selecting chunks and starting a job.
Open Data Foundry
Data Foundry is not an independent My Assets page, and there is no option to run it automatically during upload. If no chunks are available, finish document upload, parsing, and indexing first.
Run de-identification
Preview chunk content and source documents, then select one or more chunks. Use Select all to select all currently available items.
Select Start task and choose one or more data types to remove. All personal data applies every category at once.
Select Start de-identification. Each chunk creates one background job. A duplicate item is skipped when the same chunk already has a pending or running job.
The page refreshes progress and status periodically. After a job succeeds, the chunk list reloads the updated content.
Return to the document's chunks and validate the result with Hybrid Search or Playground. Restore an earlier version from chunk history if the replacement was incorrect.
Supported data types
| Type | Placeholder | Scope |
|---|---|---|
| Name | [NAME] | Natural-person names; excludes organizations, brands, departments, role titles, and honorific-only terms |
| Email address | [EMAIL] | Email addresses |
| Phone number | [PHONE] | Telephone and mobile numbers |
| Address | [ADDRESS] | Street, postal, and room or unit addresses |
| Date | [DATE] | Calendar dates, including date-of-birth-style dates |
| Location | [LOCATION] | Countries, cities, districts, and specific places |
| Credit card / financial information | [CREDIT_CARD] | Payment-card numbers and card details |
| Account information | [ACCOUNT] | Account IDs, usernames, member or student IDs, and bank account numbers |
| Biometric data | [BIOMETRIC] | Text describing fingerprint, face-template, iris, or voiceprint identifiers |
| Authentication information | [AUTH_INFO] | Passwords, passcodes, OTPs, tokens, API keys, and security answers |
Data Foundry currently processes chunk text. It does not modify the source PDF, Office file, image, or video, and it does not use a VLM to obscure faces in photographs.
Only the types selected for the job are processed. Unselected sensitive-data types are not requested for removal. For example, a phone number can remain unchanged when only Email address is selected.
Processing flow
The model must return exact contiguous strings present in the original chunk. Runtime validates types, values, and duplicates before applying replacements. A job can still succeed with zero replacements; success does not prove that the chunk contains no sensitive information.
Job states
| State | Meaning |
|---|---|
| Pending | Queued and waiting for a worker |
| Running | Loading content, calling the task model, or saving the result |
| Succeeded | Chunk, version, and index were updated |
| Failed | A model, configuration, balance, worker, or storage operation failed |
| Skipped | The chunk was empty, disappeared after re-indexing, or already had a duplicate active job |
A pending or running chunk is locked and cannot be selected again. The page polls job state about every three seconds. On startup, the worker also marks stale running jobs as failed so they do not remain stuck indefinitely.
Versions, indexing, and snapshots
Each successful job:
- Preserves the pre-job original chunk version.
- Creates a new chunk version with the
data_foundrysource. - Immediately writes the de-identified content to the current Milvus chunk and rebuilds its vector.
- Creates a new knowledge-base snapshot.
- Supersedes an older Marketplace submission that is still under review.
To restore content, open the source document's chunks, preview a version under Chunk history, then select Restore this version. Restoration also overwrites the currently searchable chunk, so test retrieval again afterward.
Data Foundry does not produce a downloadable de-identified source file. The source file remains unchanged; versioning applies to the knowledge-base chunks.
Billing
The task model is billed from actual input and output tokens:
- Personal-workspace knowledge bases use the personal wallet.
- Organization-workspace knowledge bases use the organization wallet.
- Free pricing still produces a usage record.
- When the balance is insufficient, the job fails and Studio opens the top-up prompt.
Selecting several chunks creates several jobs. Each job calls the model and is billed separately. Start with a small representative sample, validate the categories, and then expand the batch.
Limits and safety guidance
- Data Foundry is LLM-assisted best-effort detection and cannot guarantee removal of all sensitive information.
- Review a sample after processing, especially for medical, financial, identity, and regulatory data.
- Ambiguous names, unusual account formats, or values without context can be missed or misclassified.
- Jobs directly change RAG retrieval content. Before a large batch, confirm that a snapshot and versions are available for recovery.
- Request Logs and audit records can contain job input, output, model, token, and replacement information. Restrict log access according to your data policy.