Ingestion troubleshooting
PDF Failed to Ingest into Your AI Knowledge Base? Diagnose It First
Diagnose empty, skipped, malformed, oversized, or image-only PDFs before trusting an AI knowledge base or RAG answer.
Article packet
Workflows
People whose AI knowledge-base import reports success, skip, or error but produces empty, incomplete, or unusable PDF content
5 min read
01
First distinguish a text PDF from a scanned or image-only PDF.
02
Treat found, ingested, skipped, and error counts as diagnostics, not proof that usable text reached retrieval.
03
Verify one known answer and its exact source before trusting the document set.
01
Quick answer
If a PDF produces no usable knowledge, first try selecting and copying text from the file. An image-only scan needs OCR before Wenlan can ingest it; Wenlan has no OCR in v1. Next check that the PDF is at most 10 MB, is not malformed, and contains enough real text. Run `wenlan sources add <path>`, inspect the found, ingested, skipped, and error counts, then verify that a known passage can be retrieved and cited. A successful registration or batch summary alone does not prove that extraction succeeded.
Wenlan accepts `.md`, `.txt`, and text-extractable `.pdf` Sources. Its current source connector caps PDFs at 10 MB, skips image-only or near-empty extraction, reports malformed PDF parsing as an error, and continues processing the rest of a folder. It does not perform OCR in v1.
02
When this problem appears
PDF ingestion can fail loudly with a parser error or quietly with an empty text layer. The dangerous case is an import that looks complete while the knowledge base contains no useful chunks, loses layout relationships, or cannot cite the passage a user expects.
03
Diagnose the file before changing retrieval
Work from the source inward. Do not tune embeddings or prompts until you know the expected text actually entered the index.
- Classify the PDF: if text cannot be selected or copied, treat it as a scan or image-only document.
- OCR scanned pages outside Wenlan, then save a text-extractable PDF or a clean `.md` or `.txt` file. Spot-check names, dates, numbers, tables, and page order against the original.
- Check the input boundary: folder Sources accept `.md`, `.txt`, and `.pdf`; PDFs over 10 MB, unsupported extensions, hidden files, symlinks, and skipped directories do not enter the normal folder scan.
- Run `wenlan sources add <path>` and record the found, ingested, skipped, and error counts. An error suggests read or parse failure; a skip can mean unchanged content, no extractable text, or content below the minimum quality floor.
- Wait for document processing, then retrieve a distinctive sentence that definitely exists in the source. Confirm the result opens the intended file or source reference.
- Ask one answerable question and one question the document cannot answer. The first should cite supporting text; the second should remain unknown.
- If extraction is incomplete, simplify the source to Markdown or plain text and re-run the same acceptance test. Do not hide missing tables or scanned pages behind a fluent summary.
Register or resync one diagnostic source
wenlan status
wenlan sources add ~/Knowledge/pdf-diagnostic04
What to check next
OCR can introduce wrong characters, reading order, and table structure. A searchable PDF is not automatically a faithful PDF. Compare important passages with the rendered page and keep the original file authoritative.
Verify one PDF before importing the archive
Install Wenlan, add one controlled document, and prove that its text, source, and citations survive the complete ingestion path.
FAQ