July 6, 2026 · 8:14 AM
Lenny-only newsletter digest — July 6, 2026
Today’s digest covers Lenny’s public Sonnet 5 review notes: how Claire Vo turned model testing into a repeatable bench, why she mixed human and model scoring, and why model choice should be routed by task rather than crowned with one winner.
The only new publishable source item today is narrow but useful: Lenny's feed surfaced a public "How I AI" source-note post on Claire Vo's Sonnet 5 review, while Stratechery still had no new public article or Daily Update in the checked window. Lenny's July 4 Community Wisdom item was paid-only, so it is not summarized here beyond noting why it was excluded.123
AI model evaluation
Sonnet 5 review: I ran 64 generations to find out if it's worth it
Claire Vo's public notes say she built a repeatable 「How I AI Bench」 after getting tired of one-off model tests that could not be compared over time. The test put Sonnet 5 blind against Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro across PRD quality, prototype generation, agentic task completion, and agent personality.1
Three points worth keeping:
- The evaluation harness is the real product lesson. Vo says she built the bench live with Claude Code in under 45 minutes, using her stored session history as the starting material. The useful idea for product teams is not "copy this exact leaderboard"; it is to turn repeated model judgment into a reusable test bench tied to your own work.1
- The scoring mix avoids two common traps. The notes say Vo combined human "vibe" scoring at 70% with LLM-as-judge scoring at 30%, and used a local HTML scoring page to rate outputs and export JSON. That is a practical compromise: humans still judge whether the output feels usable, while the model adds consistency to repeated comparisons.1
- Model choice is task-specific, not a single crown. The public notes frame the recommendations by task: which model to use for PRDs, which for complex prototypes, and which for daily agent chat. For PMs and operators, that is the actionable takeaway: route model choice by job type before debating an overall winner.1
Source link: Read Lenny's public source notes.
Scope note: this digest uses Lenny's public source notes for the post. It does not summarize the full podcast audio or make claims about results that are only available inside the episode.
Source check
Stratechery's homepage still showed the June 29 summer-break notice, stating that there would be no Weekly Article or Updates during the week of June 29 and that the next Update would be Monday, July 6. The newest visible Stratechery Plus item remained the June 25 Dylan Field interview, so there was no new public Stratechery item to include at the time this digest was prepared.3
Lenny's default archive still showed the June 30 "How top PMs increase their leverage with AI" essay at the top, which was already covered in the July 1 issue. The feed also surfaced the July 4 Community Wisdom post, but the article page marks it as paid-only, so it does not provide enough public text for this digest's three-point format.42
One thread to watch
The important shift is from "which model feels best today?" to "how quickly can a team rerun its own evals when models change?" Vo's note is a reminder that product teams need lightweight, repeatable tests for their own workflows. A model leaderboard matters less than knowing which model reliably writes your PRDs, builds your prototypes, or behaves well as an everyday agent partner.
Related content
- Sign in to comment.