The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
arXiv preprint, 2026 · · · dataset

TL;DR: Taste-Bench measures whether LLM agents choose the better direction at real decision forks: on 502 questions mined from agent trajectories, the best of 14 frontier models answers only 59.7% correctly, and a larger reasoning budget does not help.

Each question freezes an engineering or research trajectory at a fork, hides everything after it, and asks the model which of two directions leads to the better outcome. Forks are mined without human annotation from parallel attempts at the same task and from detours inside one trajectory (4,657 candidates, 10.8% pass all filters). Accuracy falls as the deciding evidence moves later: the 14-model mean drops from 62.3% to 21.0%, near the 25% random score. Taste is also trainable: a student distilled from a hindsight teacher gains 17.9 points on unseen tasks, and its advice raises an executor’s success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%. Joint work with Microsoft Research Asia.

Abstract

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.

BibTeX

@article{pan2026tastebench,
  title={The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks},
  author={Pan, Wenbo and Liu, Zhichao and Liu, Shujie and Zeng, Jingying and Lin, Chin-Yew and Tang, Xianfeng and Lu, Yan and He, Qi and Jia, Xiaohua},
  journal={arXiv preprint arXiv:2609.25804},
  year={2026}
}

All papers are on the publications page.