Lemkin: Jev 90x cheaper, 32% false admissions
Jason Lemkin ran Jev against Sonnet on SaaStr's candidate judging: 600 judgments for $0.064 versus roughly $5. It was also wrong in a way he could not ship, and that is the trade the whole cheap-model wave keeps hiding.
Real money
Lemkin says Jev cost $0.064 for 600 judgments and still failed
Jason Lemkin spent a day testing Jev on SaaStr Connect's candidate judging and says it was roughly 90x cheaper than Sonnet on identical inputs, $0.064 for 600 judgments against about $5. Against a blind third-model referee it scored 70.5% accurate to Sonnet's 77.5%.
Jev unfortunately lets too many weak answers (candidates) through. A 32% false-admission rate is too high.
Stolen playbooks
Lemkin's fix: sweep volume with the cheap model, hand winners up
After rejecting Jev as a replacement, Lemkin says he landed on using it only additively: sweep the volume you currently skip, hand the best candidates to the expensive model. He explicitly rules out using it as a pre-filter, because a cheap model rejecting silently removes good options before anything better sees them.
not as a pre-filter either, since a cheap model rejecting silently removes good options before anything better sees them
What changed under us
Rauch says the best part of this AI wave is everyone wants to build
Guillermo Rauch, setting aside what he calls the short-term drama, says the notable thing about the current AI wave is that everyone is excited about building rather than consuming. He calls it an immeasurably good development. No numbers attached, and he does not name the drama.
The people want to create and ship, not just consume.