Yugesh B

Language breakdown (live from GitHub)

Key Contributions

• Streaming chat engine with multi-key API failover — round-robin Gemini API key rotator with quota-aware failover, exponential backoff, and auto-continue logic that transparently resumes responses cut off by output-token limits.
• Self-refreshing legal dataset via GitHub Actions — a daily cron workflow pulls new Supreme Court, High Court, and tribunal judgments from Indian Kanoon's official RSS feeds and auto-commits new entries with no manual upkeep.
• Offline regex-based Q&A extraction pipeline — parses raw statutory text directly into 20,000+ structured Q&A pairs with no LLM calls, plus a secondary LLM-assisted extraction path for judgment text.
• Multi-source legal web scraping across three independent sources, each normalized into one unified Q&A schema.

Key Contributions

• Config-driven scraping framework processing 229 distinct site sources from a single unified codebase, eliminating per-site custom scripts via declarative YAML configuration.
• Multi-engine fallback scraping chain (Selenium → Playwright → static HTTP) that escalates to heavier browser automation only when a lighter engine fails.
• Dual-database persistence layer (MySQL + ClickHouse) with automated staging tables, cross-database sync, and a date-validation quarantine system for ambiguous or invalid dates.
• LLM-based fallback date-resolution mechanism (via Ollama) as a last-resort recovery step before any record is quarantined.

Key Contributions

• Broker-free distributed job queue in MySQL using row-level locking (SELECT ... FOR UPDATE SKIP LOCKED) so concurrent workers atomically claim eligible work with no message broker and no race conditions.
• FastAPI control plane plus an APScheduler-based central scheduler reading cron/interval schedules from the database as the single source of truth for 'is this job due.'
• Zero-code-duplication architecture — one central code server exposes all ~1,500 scripts over a live SMB network share (via Windows symbolic links, not junctions, since junctions don't support network paths); new worker capacity needs only a bare Python venv.
• Full observability: every execution attempt is recorded with status, timing, and — on failure — the complete traceback plus a classified error type (dependency, timeout, network, HTTP 403/429, browser-automation).

Key Contributions

• Config-driven core (table_registry.yaml) declaring every one of 28 source tables once — its S3 landing path, target ClickHouse table, field overrides, and schedule — so adding a new source is a config entry, not new code.
• Generic chunking engine: downloads from S3, extracts text (pypdfium2 + PyMuPDF with scanned-page/OCR detection), classifies document type, chunks and embeds via VoyageAI, and writes to ClickHouse with safeguards so a partial failure never marks a truncated document 'done.'
• Aggregation layer rolling up 28 per-source chunk tables into 7 shared collection tables that downstream retrieval actually queries against.
• Diagnosed a silent production failure where 'chunked' rows never processed further — traced through the full trigger chain to a broken n8n integration executing commands on the n8n host instead of inside the application container, affecting every scheduled table.

Key Contributions

• Reverse-engineered the portal's undocumented JSON API (maker-report-page), identifying the exact request shape required and confirming the endpoint is captcha-exempt within a session's trust window — bypassing the need to click through the UI for every query.
• Built two parallel scraping engines: a Playwright browser-automation engine as ground truth, and a pure-requests engine talking to the JSON API directly — 5–10x faster and verified via live A/B testing to mirror the browser engine exactly.
• Automated captcha solving via local (Ollama) and cloud vision-language models, with magic-byte image validation and a 3-attempt retry loop so a single OCR misread doesn't silently drop a data combination.
• Resilience layer: proxy fallback (Zyte) triggered only on genuine blocking codes (403/429/503, distinguished from real portal downtime), 3-way concurrent state scraping, per-combination failure tracking with automatic end-of-run retry, and per-state resumability after a killed run.

Key Contributions

• Pipeline: S3-sourced PDFs → PyMuPDF-based page/table classification (regex + keyword heuristics) → coordinate-based (bounding-box) table reconstruction → two-pass LLM+VLM extraction → cleaning/normalization → ClickHouse/MySQL storage → automated validation → Excel reporting.
• Deterministic coordinate-based table parsing — for complex multi-segment tables (e.g. Fire/Marine/Motor/Health columns × quarter/YTD sub-columns), clusters raw word bounding boxes into rows/columns from geometry alone, before ever calling an LLM.
• Two-pass multimodal extraction: an LLM text pass drafts values from parsed PDF text, then a VLM vision pass (Gemini / Ollama-hosted Gemma, page rendered as a 300–600 DPI image) visually cross-checks row/column alignment and figures against the source layout — catching misalignments pure text extraction misses.
• Engineered a custom compact output format (TOON — Table-Oriented Output Notation) and parser to cut LLM/VLM token usage and simplify downstream parsing versus verbose JSON/CSV.