ENG-028deployed

Lead Intelligence System — rubric-first lead qualification

A sales team that reads 5% of its leads is not choosing the best 5%. The fix was not a smarter model — it was an explicit rubric the team can argue with, and a model that explains, writes and flags, but never decides the number.

Client

IIT Roorkee PGC in Forward Deployed AI Engineering (Cohort 2) — mini project, September 2026. Simulated client: a mid-market SaaS vendor with ~1,200 inbound leads a month.

Role

Solo build, AI-assisted. I owned the problem structuring, the rubric and its weights, the failure design, the build and the write-up.

Timeframe

September 2026 — a one-evening build against a fixed submission deadline

Stack

Python 3 · LangChain (LCEL) · Gemini 2.5 Flash · YAML-driven rubric · Batched LLM calls · Exponential backoff · JSON + CSV reports

◇ source → devaramanelohith/lead-intelligence-system

Command-line tool, so no hosted demo. The repository ships the three academy datasets, the full 100-lead output report, and an implementation guide with diagrams so the build can be reproduced phase by phase.

Screens

Six-stage pipeline: intake, rubric, decision, LLM batch, merge, report, with the deterministic stages bracketed separately from the LLM stage — The shape of the system. Stages two and three are plain Python and produce the same score on every run. Stage four is the only place a language model appears, and it receives the score rather than producing it. The red path is the design for a bad night: if Gemini is unreachable after five tries, the affected leads get rubric-generated text and a flag, and the run finishes anyway.

Five scoring factors with their point bands, the override rules, the decision thresholds, and a worked example for one lead — How a lead is classified. Five factors add to ten points; every band is a line in config.yaml, not a constant in code. Override rules run first so that a stale lead is rejected however good it looks on paper, a lead with two missing fields is never auto-qualified, and a two-person startup with a perfect intent signal goes to a human rather than to the top of the queue.

Terminal summary of a 100-lead run showing retries, the decision split and the top of the priority queue, plus one lead's JSON record — A real run on the 100-lead file: five batches, eight Gemini calls, three of them automatic retries on free-tier rate limits, zero fallbacks. Below, one lead's record: the factor breakdown that produced the score, the model's reasoning that cites those factors, its own recommendation, and the drafted first message.

01 — Situation

The brief described a mid-market SaaS company receiving about 1,200 inbound leads a month. Each one was judged by hand, eight to twelve minutes a decision, so the team reached roughly sixty. The other 95% were not rejected; they were simply never read. The company suspected it was both missing good leads and spending its scarce rep time on poor fits, and had no way to tell which.

The academy supplied three datasets — thirty clean leads for development, a hundred for the production-scale run, and twenty deliberately broken rows — and, pointedly, no answer key. The assignment was to define what 'qualified' means, apply it consistently at scale, defend every decision, and produce a report a sales team could act on the same morning.

02 — Constraints

!No ground truth. There is no conversion history to learn from, so the qualification logic had to be explicit, defensible and consistent rather than accurate against a label that does not exist.
!An LLM API was mandatory, but calling it once per lead was explicitly forbidden — the system had to batch, and it had to survive rate limits, timeouts and malformed output without a crash.
!Messy input by design: blank fields, 'NA' strings, a one-person company, a 50,000-person enterprise, a last contact from 2022, and a row with nothing but a name.
!A free-tier model with strict requests-per-minute limits, and one evening between receiving the brief and the submission deadline.

03 — Stakeholders

Sales rep

A ranked list to work top-down, and a message that already knows who the lead is

Sales manager

A rubric they can read, argue with and change without asking an engineer

The borderline lead

Being looked at by a person, not silently guessed into a bucket

Academy evaluators

Consistency and explainability of every decision, and honest handling of failure

04 — Architecture

01Six stages in one command: intake, rubric scoring, decision, LLM batch, merge, report. The first three are deterministic Python. The same CSV produces the same scores on every run, which is what makes the decisions auditable.
02The rubric is five factors adding to ten points — company-size fit, industry fit, buying intent inferred from the lead source, recency of last contact, and data completeness — with every band and threshold in a YAML file. A manager can move a number and re-run; no code changes.
03Override rules run before thresholds: a lead untouched for over a year is rejected regardless of score, a lead missing two critical fields goes to review, and unknown categories or extreme company sizes are capped so they land in front of a human. The review bucket is a feature, not a leak.
04The model's job is bounded. A LangChain chain — prompt template, Gemini 2.5 Flash, JSON output parser — receives twenty leads per call together with their scores and factor breakdowns, and returns for each a reasoning paragraph that cites the factors, a second-opinion recommendation, and a personalised first message in one of three styles chosen by source. It is told, in the prompt, that it may not change the score.
05Failure is designed, not handled: retries with exponential backoff on rate limits and server errors, a JSON parser that tolerates fences and wrapped arrays, a normaliser for the model's habit of answering 'REVIEW, leaning REJECTED', and a fallback that writes rubric-generated reasoning and a template message for any lead the model did not return — flagged, logged, run continues. A missing API key drops the whole run to rubric-only mode with a warning.

05 — What shipped

✓A command-line tool that processes the 100-lead file end to end in about five minutes with no manual steps, and the 20-row adversarial file with zero crashes — every incomplete or extreme lead lands in review with a flag saying why.
✓A JSON and CSV report: per-lead score, factor breakdown, decision, priority rank, reasoning, flags and outreach message; aggregate stats with top rejection reasons; a ranked priority queue; a review queue with the model's lean; five sample messages across sources.
✓Personalised outreach for every qualified lead, naming the contact, company, industry and how they engaged — a demo follow-up with time slots, a warm referral note, or a soft re-engagement after a webinar or download.
✓A one-page README with the rubric in three sentences, and an implementation guide with architecture and decision diagrams so the build can be repeated phase by phase.

06 — Outcomes

100

leads qualified, ranked and messaged in one run — ~17 rep-hours at the manual rate

crashes across 149 leads including 20 deliberately broken rows

3 / 0

rate-limit retries absorbed / fallbacks needed on the final run

07 — Retro: what I'd do differently

The first rubric qualified 85% of the file. That number was the most useful bug of the evening: a team that currently pursues 5% would not trust a tool that says 'almost everyone'. Lowering the weight of low-intent sources and non-target industries and raising the thresholds brought it to 58% qualified, 26% review, 16% rejected — and made the review bucket meaningful.

Two leads exposed a rule I had not written. A two-person startup with a same-day demo request and a 50,000-person enterprise both scored HIGH on raw points. Neither belongs at the top of a mid-market queue without a human looking, so size extremes now cap the score into review. The lesson generalises: the rubric is where judgment lives, and every surprise in the output is a rule waiting to be made explicit.

Keeping the model away from the score was the right call, and the second-opinion field earned its place — the model disagreed with the rubric on nine of a hundred leads, all of them in the review bucket, and its stated lean is exactly what a rep wants to read before deciding.

Next, in order: an evaluation set for message quality scored for personalisation and tone, since a message is the one output a customer actually sees; conversion feedback so the weights can be fitted instead of hand-set; and a CRM adapter so the priority queue lands where reps already work.

Questions about this engagement?

Ask the copilot →