@mayaanderssondev
Just a bored curious dev
Nothing here yet.
Nothing here yet.
23h ago · 4 min read · TL;DR: I ran six prompt-optimization frameworks against the same task and the same eval metric over a few weeks. They are not interchangeable: some are full programming models, some are single search
Join discussion
3d ago · 3 min read · TL;DR: I compared the main LLM-as-judge tools (DeepEval's G-Eval, Confident AI, Evidently, Braintrust, Promptfoo, and MLflow) on the axis that actually decides whether the scores mean anything: how we
Join discussion
3d ago · 3 min read · TL;DR: A headline eval pass rate is an average over every kind of input your system sees, and averages hide the thing you most need to catch: a sharp regression in a small but important slice. If refu
Join discussion
Jun 11 · 2 min read · We track weekly agreement between an LLM judge and human labels (Cohen's kappa) on a sample of production traces. For three weeks the point estimates told a story: 0.55, then 0.49, then 0.44. The team
Join discussion
Jun 9 · 3 min read · TL;DR: Our LLM-as-judge agreement (Cohen's kappa against human labels) swung between 0.41 and 0.63 week to week with no rubric change. First instinct was sample size, so we went from 50 weekly traces
Join discussion