Neal Desai

Forward Deployed · GTM · Product · Engineering

GH LI
01 EscalationBenchResearch preview EscalationBench measures whether AI agents recognize when context, authority, or policy is insufficient and stop to ask, verify, or hand off when they should. It scores how well they calibrate that judgment, including over-asking and under-asking, across real business workflows beyond code and SQL. Read the writeup →
02 Manufacturing Synthetic Evaluation DataWriteup A methodology for generating benchmark data that holds up: one shared pipeline, a recipe per data type, difficulty measured by a solver panel, a review layer of quality gates, and a human verdict on every point. Written to be adapted to your own evaluation format. Read the writeup →
03 Loop Engineering for Agent SkillsWriteup A skill only pays off when the agent invokes it at the right moment. A methodology for measuring skill triggering as a compliance rate: synthesize positives and hard negatives per skill, run them through Claude Code, Codex, or your own harness, score invocations straight from the tool log, and raise the curve with a /goal prompt and targeted edits. Skills are the worked example; the same loop hill-climbs any behavior you can quantitatively measure. Read the writeup →
04 The DispatchEdition 2 live A recurring read of the technical-build signal from the AI talks worth watching. A pipeline ranks the fortnight's talks by trend signal, transcribes them, extracts and scores the founder takeaways worth acting on with Claude, and synthesizes a dated edition. The short version leads with three or four picks; every full edition stays archived, and each featured takeaway pops out a player at the exact source moment. Read the latest edition →

I'm a Staff Strategic Product Manager at Scale AI, a pre-sales role working with research teams at frontier AI labs on reinforcement-learning environments, fine-tuning data strategy, and model evaluations. Much of the work is scoping and building data products and the unique insights that come with them, from studying where models fall short and closing the gap.

Before Scale I was a startup co-founder and CTO, built production ML at Beyond Limits for enterprise customers in energy, finance, and healthcare, and started out at Epic Systems shipping predictive models for hospitals. I studied biological engineering and applied math at Caltech and machine learning at UW-Madison. I'm based in New York. Outside of work I mentor high school students through their first real research projects in healthcare and tech, which is some of the most rewarding work I do. The rest of the time you'll usually find me traveling or outdoors.

Have an idea, a question, or just want to say hi? I'd love to hear from you. Reach me on LinkedIn or by email.