10/15/2026
I spent the last couple of days putting a new AI decision model through its paces against the locally hosted LLM auditors we use at ProcureVue.
The initial results looked impressive: adding one domain-specific rule took recall from 15% to 89%, eventually reaching 92% at 98% precision. But digging into the results told a very different story. That dramatic improvement was driven largely by one product family dominating the test data. When I moved to a second client's actual record-linkage pipeline output, our existing local auditors proved more effective at what matters most to us: finding legitimate matches without overwhelming human reviewers.
Along the way, I uncovered some interesting lessons about domain-specific rules, fine-tuned versus zero-shot models, evaluation bias, and why precision and recall alone don't tell you which system belongs in production.
The entire experiment cost $16.16 in API charges. I wrote up the methodology, results, failures, and a few surprises in a new Medium field report.
https://pub.towardsai.net/can-a-local-auditor-ensemble-match-a-hosted-decision-model-48faac0f5111
I wrote five domain rules for a hosted decision model and tested it against my locally hosted LLM auditors. On my development data, where…