DOCUMENT AI · OCRSole engineer · Deacons Publishers (Prep50)
Printed page citations from image-only scans
Recovering the printed page number for 83,807 textbook sections after OCR had discarded page positions and the PDFs turned out to have no text layer.
- 99.3%
- sections with a page citation
- 93 of 102
- books citing printed page numbers
- 24.9 h
- unattended run, 0 failed books
- ~$32
- total cost, on AWS credit
Generated study notes needed to cite a real page, like “Comprehensive Biology, p. 142”. The data for that did not exist: the OCR worker had thrown away page indexes, and the scans were pure image containers with nothing to extract. The approach was to OCR a strip at the bottom of each page, use its last line of text to find where the page ends in the book’s Markdown, and read the printed page number from the same strip. Books whose numbers could not be read fall back to an honest scan position instead of a guessed page.
WHAT BROKE, AND WHAT I DID ABOUT IT
The first crop contained no text at all
I set the strip to 12% of page height without measuring, and 0 of 56 test pages anchored. An ink profile showed the bottom 15% of these pages is blank margin; the only mark in frame was a pen scribble on the scanner bed. Measuring first and moving the strip to 28% took anchoring to 80%.
Repeated headings outvoted the right answer
A practical physics book has 89 sections called “Method”. Two stray text matches far down the book were beating a correct cluster of 70 matches. Scoring clusters by support before position cut dropped anchors from 51 to 2, and a longest increasing subsequence stopped one bad anchor from blocking every page after it.
Zero errors was hiding a 15x slowdown
Throughput sat at 0.20 pages/s with no errors, because my retries were quietly turning throttling into waiting. Measuring showed Bedrock limits are per account per region, so a second instance would have made both slower. A cross-region inference profile reached 1.33 pages/s and cut mean latency from 22.3s to 4.7s.
The trust test used the wrong measure
I gated printed numbers on the share of readings agreeing on one offset. It failed on science books, where a page’s last token is often a quantity: a readable chemistry book scored 0.466. Adding a dominance test (24:1 for that book, 1.5:1 for a genuinely unreadable one) fixed it. OCR and matching were separate stages, so the fix took half an hour instead of another 25-hour run.
STACK
- Python
- AWS Bedrock
- EC2
- S3
- pypdf
- PostgreSQL 17
- Concurrency & backoff
DATA ENGINEERING · DOCUMENT AISole engineer · Deacons Publishers (Prep50) · 2025 to present
From exam paper to vector index
A six-stage pipeline turning two decades of WAEC and JAMB papers, from printed booklets to screenshots to inconsistent Word documents, into a classified, queryable corpus.
- 55,106
- questions in production
- 11,000+
- extracted and verified
- 3,730
- learning objectives
- 12
- subjects covered
An exam-prep product is only as good as its question bank, and this one existed as paper: scanned booklets, tutor videos recorded off a projector, and Word documents typed by a dozen people across fifteen years. The interesting engineering is not the AI. It is everything built around the AI so that when it is wrong, and it is regularly wrong, nothing downstream silently breaks.
WHAT BROKE, AND WHAT I DID ABOUT IT
A one-character typo hid 793 questions
The uploader wrote to a prep50/ prefix; every downstream filter tested for pre50/. Nothing errored, no test failed, every run reported success, and 793 verified questions were quietly skipped for months. A filter clause that matches zero rows is now treated as a failure, not a valid result.
Two databases cannot share a transaction
Migration writes to production Postgres, then records the ID back in MySQL. Instead of pretending that gap does not exist, the Postgres insert and its objective link share one transaction, and a failure on the second write journals the new ID to a recovery file with enough context to reconcile by hand. Loud, bounded and repairable beats silent.
Knowing when not to call the model
Mathematics documents already carried human-authored subtopics, so sending them to an LLM would have been slower, costlier and less accurate than the people who tagged them. The classifier takes a direct path when an objective already exists: 425 of 445 questions on the last run matched production taxonomy exactly, at zero API cost.
Every destructive operation is a plan and an apply
Nothing mutates data in one step. The taxonomy cleanup planner is strictly read-only. It normalizes, groups duplicates, picks a canonical row by actual usage and writes a reviewable plan, flagging the risky cases. Only then does the applier run. It retired 202 objective IDs, 3 of which needed human judgement.
STACK
- Python
- GPT-4o vision
- Gemini 2.5 Pro
- EasyOCR
- OpenCV
- PyMuPDF
- PostgreSQL 17
- pgvector
- MySQL 8
- FastAPI
- Docker
ML INFRASTRUCTURE · AWSSole engineer · Deacons Publishers (Prep50)
Textbook digitization pipeline
An event-driven GPU pipeline turning 102 scanned curriculum textbooks into clean, structured Markdown, unattended, recovering from the failure modes of open-source ML inference at scale.
- 102
- textbooks digitized
- 31,887
- pages processed
- A10G
- GPU worker, 24GB VRAM
- 0
- recurrences after the fix
Scanned exam-prep textbooks, dense with tables, multi-column layouts and diagrams, that off-the-shelf OCR mangled badly enough to be unusable. A PDF leaves a field laptop, lands in S3, fires an SQS job, and a GPU worker returns structured Markdown with images preserved. No polling, no manual triggering, least-privilege IAM scoped per component.
WHAT BROKE, AND WHAT I DID ABOUT IT
Poison pages sent the model into runaway generation
Dense answer-key pages hung the VLM indefinitely. A live py-spy stack trace pinned it inside HuggingFace's _calc_banned_ngram_tokens, whose cost scales with what has already been generated. The fix kills the whole process group on a page-count-scaled timeout, because MinerU spawns its own child server and a naive kill would orphan it still holding GPU memory.
A leak that only appeared after days of uptime
A wave of unexplained crashes with nothing in dmesg or the kernel journal. The real signal was buried in the model server's own subprocess output: CUDA out of memory from VRAM fragmentation accumulating across hundreds of sequential invocations. One allocator setting fixed it, verified across a full backlog re-run.
Rejecting the clever fix on purpose
Recursively bisecting a failing chunk to isolate the exact bad page worked, and cost 45 to 75 minutes per book, because every bisection level pays its own full timeout. Against a 90-book backlog that is GPU time stolen from 89 other books. Fail-fast to a dead-letter queue won, and the rejected approach is documented so it does not get silently re-proposed.
Graceful degradation instead of total loss
A chunk timeout used to cost the entire remaining book. Now it drops to one page at a time under a smaller timeout and skips only the pages that are genuinely unprocessable, marking each one as a red callout in the viewer so QA goes straight to it. Books that would have been a total loss now land complete with a handful of flagged pages. One hit this three times and still produced 372 files.
STACK
- Python
- Bash
- AWS EC2
- S3
- SQS + DLQ
- IAM
- CloudWatch
- systemd
- MinerU
- PyTorch / CUDA
- Flask
- py-spy
