DATA ENGINEERING · DOCUMENT AISole engineer · ~10 months, ongoing
From exam paper to vector index
A six-stage pipeline turning two decades of WAEC and JAMB papers, from printed booklets to screenshots to inconsistent Word documents, into a classified, queryable corpus.
- 55,106
- questions in production
- 11,000+
- extracted and verified
- 3,730
- learning objectives
- 12
- subjects covered
An exam-prep product is only as good as its question bank, and this one existed as paper: scanned booklets, tutor videos recorded off a projector, and Word documents typed by a dozen people across fifteen years. The interesting engineering is not the AI. It is everything built around the AI so that when it is wrong, and it is regularly wrong, nothing downstream silently breaks.
WHAT BROKE, AND WHAT I DID ABOUT IT
A one-character typo hid 793 questions
The uploader wrote to a prep50/ prefix; every downstream filter tested for pre50/. Nothing errored, no test failed, every run reported success, and 793 verified questions were quietly skipped for months. A filter clause that matches zero rows is now treated as a failure, not a valid result.
Two databases cannot share a transaction
Migration writes to production Postgres, then records the ID back in MySQL. Instead of pretending that gap does not exist, the Postgres insert and its objective link share one transaction, and a failure on the second write journals the new ID to a recovery file with enough context to reconcile by hand. Loud, bounded and repairable beats silent.
Knowing when not to call the model
Mathematics documents already carried human-authored subtopics, so sending them to an LLM would have been slower, costlier and less accurate than the people who tagged them. The classifier takes a direct path when an objective already exists: 425 of 445 questions on the last run matched production taxonomy exactly, at zero API cost.
Every destructive operation is a plan and an apply
Nothing mutates data in one step. The taxonomy cleanup planner is strictly read-only. It normalizes, groups duplicates, picks a canonical row by actual usage and writes a reviewable plan, flagging the risky cases. Only then does the applier run. It retired 202 objective IDs, 3 of which needed human judgement.
STACK
- Python
- GPT-4o vision
- Gemini 2.5 Pro
- EasyOCR
- OpenCV
- PyMuPDF
- PostgreSQL 17
- pgvector
- MySQL 8
- FastAPI
- Docker
ML INFRASTRUCTURE · AWSSole engineer · design, build, operation
Textbook digitization pipeline
An event-driven GPU pipeline turning ~96 scanned curriculum textbooks into clean, structured Markdown, unattended, recovering from the failure modes of open-source ML inference at scale.
- ~96
- textbooks digitized
- ~31,000
- pages processed
- A10G
- GPU worker, 24GB VRAM
- 0
- recurrences after the fix
Scanned exam-prep textbooks, dense with tables, multi-column layouts and diagrams, that off-the-shelf OCR mangled badly enough to be unusable. A PDF leaves a field laptop, lands in S3, fires an SQS job, and a GPU worker returns structured Markdown with images preserved. No polling, no manual triggering, least-privilege IAM scoped per component.
WHAT BROKE, AND WHAT I DID ABOUT IT
Poison pages sent the model into runaway generation
Dense answer-key pages hung the VLM indefinitely. A live py-spy stack trace pinned it inside HuggingFace's _calc_banned_ngram_tokens, whose cost scales with what has already been generated. The fix kills the whole process group on a page-count-scaled timeout, because MinerU spawns its own child server and a naive kill would orphan it still holding GPU memory.
A leak that only appeared after days of uptime
A wave of unexplained crashes with nothing in dmesg or the kernel journal. The real signal was buried in the model server's own subprocess output: CUDA out of memory from VRAM fragmentation accumulating across hundreds of sequential invocations. One allocator setting fixed it, verified across a full backlog re-run.
Rejecting the clever fix on purpose
Recursively bisecting a failing chunk to isolate the exact bad page worked, and cost 45 to 75 minutes per book, because every bisection level pays its own full timeout. Against a 90-book backlog that is GPU time stolen from 89 other books. Fail-fast to a dead-letter queue won, and the rejected approach is documented so it does not get silently re-proposed.
Graceful degradation instead of total loss
A chunk timeout used to cost the entire remaining book. Now it drops to one page at a time under a smaller timeout and skips only the pages that are genuinely unprocessable, marking each one as a red callout in the viewer so QA goes straight to it. Books that would have been a total loss now land complete with a handful of flagged pages. One hit this three times and still produced 372 files.
STACK
- Python
- Bash
- AWS EC2
- S3
- SQS + DLQ
- IAM
- CloudWatch
- systemd
- MinerU
- PyTorch / CUDA
- Flask
- py-spy
