AI agents are becoming a promising new research direction with potential applications in the real world. These agents use foundation models such as large language models (LLMs) and vision language models (VLMs) to take natural language instructions and pursue complex goals autonomously or semi-autonomously. However, a recent analysis by researchers at Princeton University has revealed several shortcomings in current agent benchmarks and evaluation practices that hinder their usefulness in real-world applications. One major issue the researchers highlight in their study is the lack of cost control in agent evaluations. AI agents can be much more expensive to run than a single model call, as they often rely on stochastic language models that can produce different results when given the same query multiple times. In practical applications, there is a limit to the budget available for each query, making it crucial for agent evaluations to be cost-controlled.
Joint Optimization for Accuracy and Inference Costs
To increase accuracy, some agentic systems generate several responses and use mechanisms like voting or external verification tools to choose the best answer. Sometimes sampling hundreds or thousands of responses can increase the agent’s accuracy, but at a significant computational cost. Failing to control for costs in agent evaluations may encourage researchers to develop extremely costly agents simply to top the leaderboard. The Princeton researchers propose visualizing evaluation results as a Pareto curve of accuracy and inference cost and using techniques that jointly optimize the agent for these two metrics. The researchers evaluated accuracy-cost tradeoffs of different prompting techniques and agentic patterns introduced in different papers. Joint optimization can lead to the development of agents that cost less while maintaining accuracy, enabling researchers and developers to strike an optimal balance between accuracy and inference costs.
Another issue highlighted by the researchers is the difference between evaluating models for research purposes and developing downstream applications. In research, accuracy is often the primary focus, with inference costs being largely ignored. However, when developing real-world applications on AI agents, inference costs play a crucial role in deciding which model and technique to use. Evaluating inference costs for AI agents is challenging due to varying costs from different model providers and changing costs of API calls. The researchers emphasize the need for benchmarks to address this issue and provide accurate comparisons based on token pricing to evaluate models based on real-world inference costs.
The researchers found overfitting to be a serious problem for agent benchmarks, as they tend to be small and can allow agents to take shortcuts, even unintentionally. Benchmark developers should create holdout test sets that are composed of examples that can’t be memorized during training to address this problem. Many agent benchmarks lack proper holdout datasets, allowing agents to take shortcuts and provide inflated accuracy estimates. Different types of holdout samples are needed based on the desired level of generality of the task that the agent accomplishes, ensuring that shortcuts are impossible for agents. Benchmark developers have the responsibility to design benchmarks that prevent overfitting and shortcut taking by agents.
Testing the Limits of AI Agents
With AI agents being a new field, the research and developer communities have yet much to learn about how to test the limits of these new systems that might soon become an important part of everyday applications. AI agent benchmarking is new, and best practices haven’t been established, making it difficult to distinguish genuine advances from hype. As the field of AI agents continues to evolve, it is crucial for researchers and developers to address the challenges posed by current agent benchmarking practices and work towards establishing reliable evaluation methods that truly reflect an agent’s capabilities in real-world applications.
Leave a Reply