Every background check at the verification platform I work on needs a reference: someone who managed the candidate has to confirm they worked there. For a while, every one of those started with a person. When an old script-and-forms setup was switched off, ops began each check by hand, emailed the former manager, read every reply, and raised every follow-up issue themselves.
In the first week of September that was 116 manual tasks a day just to start checks. By mid-October it was 14.
What I built
The work is not clever. It is a set of small, boring pieces that each take one manual step away.
- A queue that starts the check. When a check needs a reference, a BullMQ job creates the outreach instead of waiting for someone to notice. If nobody replies, the same queue follows up every 24 hours.
- A form instead of an email thread. Managers used to answer in Google Forms and email tables. Now they get a short form inside the product, so their answer lands as data, not as text someone has to copy.
- An inbox for the replies that still come by email. Some managers reply to the email anyway. Those land in a threaded inbox in the portal, over AWS SES, which also let ops retire a shared mailbox that kept locking them out.
- An LLM that reads each reply and routes it. A model on AWS Bedrock sorts every reply into one of 8 labels and sends it to grading, clearance or a follow-up. The rule that matters: when it is not sure, it does not guess. The reply goes to a person.
- Automatic follow-up issues. No reply, wrong contact or an opt-out used to be raised by hand. Now the system raises them.
What changed
Comparing a manual week (1-9 September) with a week after the rollout (8-16 October), from production data:
- 86% of checks now start automatically: 2,838 automatic against 818 manual.
- Manual initiation tasks fell from 116 a day to 14.
- Average turnaround dropped from 22.4 to 15.4 days, about 31% faster, while completed checks per 9 days rose from 761 to 822.
- Manual actions per completed check fell from 6.3 to 4.4.
- The share of tasks closed by the system went from 10% to 63%.
- 85% of manager responses now arrive through the new form.
- Campaigns stuck in review went from 317 to 10.
What I would tell another team
Start with the step people hate most, not the most interesting one. The LLM is the part people ask about, but starting checks automatically removed more work than anything else.
Make the uncertain path explicit. A classifier that is right most of the time is only useful if the cases it is unsure about have somewhere to go. Ours go to a person, and that is what made ops comfortable letting it run.
Measure the before. I could only say “116 to 14” because we kept a manual baseline week. Without it, this would be a feeling, not a number.
Stack: Node.js, Express, MySQL, BullMQ, Redis, AWS SES, AWS Bedrock, React and TypeScript.
If a person on your team starts, chases or reads something by hand every day, it is probably the same shape of problem. Tell me about it.