Benchmarks

We release three datasets: 28,045 real terminal input-output pairs, a 1,000-session multi-turn interaction dataset, and a 25,849-instance labeled classification dataset.

The introduced resources are for terminal command simulation and interactive command-language generation, not data analysis or participant model-building tasks.

Terminal simulation, framed as a terminal command-level Turing test, is a long-standing problem of symbolic language generation in dialogue and interactive systems.

we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage

AgencyBench evaluates broad autonomous agent capabilities in real-world scenarios with deliverables and rubrics, without metadata specifying data analysis workflows or participant model-building tasks.

we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics.

We introduce AnalystBench, a continually extensible benchmark of 20 real-world report generation tasks grounded in multimodal document collections

AnalystBench tasks require evaluated systems to synthesize large multimodal document collections into professional reports, which is a data-grounded analysis/reporting deliverable rather than model building.

We introduce AnalystBench, a continually extensible benchmark of 20 real-world report generation tasks grounded in multimodal document collections, where models must process millions of input tokens to produce long-form professional reports.

We introduce Apeiron, a scalable and extensible framework for addressing amorphous user demands through autonomous, full-lifecycle application synthesis.

The introduced resource is for autonomous application synthesis/app development, not data analysis or participant model-building for a predictive task.

We introduce Apeiron, a scalable and extensible framework for addressing amorphous user demands through autonomous, full-lifecycle application synthesis.

We also propose an evaluation protocol for dynamic PKGs that measures type/key validity and consolidation quality

The introduced evaluation protocol evaluates product-attribute knowledge graph construction quality, which is neither data analysis nor participant model-building/MLE work.

We also propose an evaluation protocol for dynamic PKGs that measures type/key validity and consolidation quality, as well as edge-level accuracy for value assertions after canonicalization.

We introduce NEWSAGENT, a benchmark for evaluating how agents search raw materials, select relevant information, and iteratively revise drafts through core journalistic functions.

NEWSAGENT evaluates journalistic search, retrieval, and news article generation rather than data analysis workflows or model-building by the participant agent/user.

Given a writing instruction and partial firsthand materials, agents must identify narrative perspectives, issue keyword-based queries, retrieve historical context, and generate complete news articles.

construct a comprehensive corpus of 18,438 pairwise comparisons

The introduced corpus supports predicting preferences between solution pairs before execution, rather than requiring the evaluated participant to perform data analysis or build/tune predictive models.

we formalize the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons.

We further present CLEF, a Cognitive Load Evaluation Framework, and curate a new benchmark from Our World in Data (OWID).

The introduced evaluation framework and benchmark evaluate deep research report generation from OWID data, with outputs compared to professional analysts’ reports, which supports a data-grounded analytical reporting task rather than model building.

generating reports comparable to professional analysts’ outputs

We present DAC-Bench, a decision-aware benchmark with compositional tasks comprising 830 episodes and 11,345 action steps across 35 applications on Android and iOS.

DAC-Bench evaluates mobile GUI agents performing compositional app interaction tasks, not data analysis workflows or participant-built predictive models.

We present DAC-Bench, a decision-aware benchmark with compositional tasks comprising 830 episodes and 11,345 action steps across 35 applications on Android and iOS.

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in data science.

The metadata says DataSciBench evaluates LLMs on broad “data science” prompts, but does not establish whether evaluated participants must perform data analysis, model-building, both, or some other concrete task.

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in data science.

we propose **DREAM** (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic

DREAM evaluates deep research agents that deliver analyst-grade reports, with no evidence that evaluated participants must perform data analysis workflows or build predictive models as the resource task itself.

Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality.

we propose a framework to evaluate the way collaboration settings may produce different research outcomes across a variety of interactive systems

The introduced evaluation framework concerns interactive systems supporting qualitative data analysis, where evaluated outcomes are research analysis outputs rather than model-building tasks.

NLP-assisted solutions to support qualitative data analysis have gained considerable traction.

We further introduce four high-stakes, edge-deployable negotiation benchmarks across debt, healthcare, emergency response, and educational domains.

The introduced resources evaluate high-stakes negotiation behavior rather than data analysis workflows or participant-built predictive models.

We further introduce four high-stakes, edge-deployable negotiation benchmarks across debt, healthcare, emergency response, and educational domains.

Our approach distills golden subgraphs via a specialized data synthesis pipeline to formulate dense rewards, facilitating a two-stage training scheme that effectively learns the interactive graph exploration policy.

The introduced resource supports complex graph-based QA through textual graph retrieval, not participant-delivered data analysis or model-building tasks despite the authors training a retriever with it कजټلتسبابونهڅژړیاမန္မာစာहिन्दीবাংলাಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡקನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡקನ್ನಡקನ್ನಡಕನ್ನಡקನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡകನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡקನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡકನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡકನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡկನ್ನಡկನ್ನಡಕನ್ನಡಕನ್ನಡկನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡկನ್ನಡکನ್ನಡಕನ್ನಡկನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡકನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡકನ್ನಡકನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡકನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡકನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡકನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡકನ್ನಡکನ್ನಡکನ್ನಡકನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡකನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡקನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡקನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡקನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡկನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡקನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡקನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡකನ್ನಡکನ್ನಡಕನ್ನಡකನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡකನ್ನಡકನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡکನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡಕನ್ನಡکನ್ನಡಕನ್ನಡ

Integrating textual graphs into Large Language Models (LLMs) is promising for complex graph-based QA.

we propose as the core of a standardized benchmark for evaluating tutor robustness

The introduced benchmark evaluates LLM tutors' robustness against adversarial student attempts to elicit answers, which is neither data analysis nor model-building by the evaluated participant.

we propose as the core of a standardized benchmark for evaluating tutor robustness

we introduce FinMRAGBench , a comprehensive multi-modal financial RAG benchmark

FinMRAGBench evaluates participants on answering expert-verified questions across representative financial analysis tasks over real-world annual reports, with no model-building task assigned to participants.

comprising 887 expert-verified QA pairs spanning five representative financial analysis tasks

we construct FormalPhysics, a dataset of 200 university-level (LaTeX) physics problems and solutions

The introduced resources support autoformalisation and theorem proving tasks, where participants deliver formal Lean representations/proofs rather than data analysis insights or trained predictive models.

We evaluate open-source models and proprietary systems on a statement autoformalisation task on our dataset via zero-shot prompting, self-refinement with error feedback, and a novel multi-stage agentic approach, and explore autoformalisation limitations in modern LLM-based approaches.

we construct LessonBench, 40 expert-designed lessons paired with two to five heterogeneous source documents

LessonBench evaluates lesson generation/teaching output quality, not data analysis workflows or participant-built predictive models under the given metadata and prior annotation context.

we construct LessonBench, 40 expert-designed lessons paired with two to five heterogeneous source documents, on which TeachCraft achieves 67.8% win rate in human evaluation and 79.6% in LLM-based evaluation against eight baselines

GCA-DS, a curated multimodal dataset grounded in the GCC states

The introduced framework/dataset supports climate analysis with geospatial processing, derived indices, and interpretable visualizations rather than asking participants to build predictive models.

Gulf Climate Agent (GCA), a tool-augmented agent for climate analysis

We present Graph Explorer, a fully automatic data synthesis pipeline that turns expert SPARQL into executable, visibility-grounded (actions may use only IDs shown in the prompt) tool supervision without manual trace labeling.

Graph Explorer supports KGQA agents that interact with knowledge graphs via tool calls to answer knowledge-intensive questions, which is graph question answering rather than data analysis or participant model-building.

Knowledge graphs (KGs) provide explicit, auditable facts, motivating KGQA agents that interact with KGs via tool calls to reduce hallucinations.

we introduce VAF, a controlled evaluation pipeline for quantifying how webpage Visual Attribute Factors influence web-agent decision-making

VAF evaluates web agents on browsing/navigation decisions under visual UI variants, not participant data analysis or model-building tasks.

browsing interaction, where agents navigate the page via scrolling and clicking the interested item, mirroring how human users browse online

we first introduce InquireBench , a comprehensive benchmark specifically designed to evaluate mobile agents’ capabilities

The introduced benchmark evaluates mobile agents' safe interaction and proactive inquiry behavior, not data analysis or model-building work by participants.

InquireBench , a comprehensive benchmark specifically designed to evaluate mobile agents’ capabilities in safe interaction and proactive inquiry with users

we develop a data-curation pipeline to construct a new dataset named InsightEval

InsightEval evaluates agents on discovering insights through exploratory analysis over datasets, which is a data analysis task rather than model building.

To discover the latent knowledge and insights hidden within massive datasets, we need to perform deep exploratory analysis to realize their full value.

Our code, datasets and implementation details are available at https://github.com/XMUDeepLIT/LegalGraphRAG.

The introduced resource supports legal retrieval, evidence verification, and final legal judgment generation rather than data analysis workflows or model-building tasks assigned to participants.

where a Researcher retrieves candidate evidence, an Auditor rigorously verifies its validity against source documents, and an Adjudicator synthesizes the set of verified evidence to render a final judgment.

LENS first constructs a large-scale dataset by transforming Ecological Momentary Assessment (EMA) responses related to depression and anxiety symptoms into natural-language descriptions, yielding over 100,000 sensor–text QA pairs from 258 participants.

The introduced resource supports generating mental-health narratives from multimodal sensing data, which is a clinical text generation task rather than data analysis or participant model-building/MLE work assigned by the resource task itself.

we introduce LENS, a framework that aligns multimodal sensing data with language models to generate clinically grounded mental-health narratives

introduce a four-layer evaluation stack spanning safety, process, outcome, and operations

The introduced framework evaluates clinical workflow agents for tasks like diagnosis, triage, consultation, and decision support, which are neither data analysis workflows nor participant model-building tasks.

clinical and healthcare workflows, including diagnosis, triage, consultation, discharge, mental health, and EHR-linked decision support

we introduce LongCLI-Bench , a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon , realistic, sequential engineering tasks.

LongCLI-Bench evaluates agents on software engineering command-line programming tasks, not data analysis or predictive model-building.

We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring.

SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments

SYNTHAGENT's resource tasks are agentic tool-use and user-querying in simulated environments, not data analysis workflows or participant model-building tasks despite being used for training small LLMs by the paper authors later in the abstract/procedure facts are methodology rather than evaluated deliverables in the resource task classification sense). This classification is based on the task behavior described for agents, not on the authors' reinforcement learning/training use of the framework, so it falls under others rather than modeling or analysis. I have selected a contiguous excerpt that directly describes the agent deliverable/task behavior: querying users for missing information while handling underspecified synthetic tasks. The abstract also mentions 'tool-use training data and simulates complete environments', but that is the resource construction/use context, not evidence that evaluated participants must build predictive models or conduct data analysis. Confidence is 4 because the abstract clearly indicates tool-use/user-interaction tasks, though it does not provide detailed examples of individual tasks within SYNTHAGENT beyond broad agentic tool-use domains.

This compels agents to actively query users for missing details.

we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research

PaperScope evaluates agents on scientific QA involving reasoning, retrieval, summarization, and problem solving over papers, not on delivering data analysis workflows or building predictive models.

It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning.

We introduce , a configurable environment that evaluates both what agents accomplish and how they interact.

The introduced environment evaluates human-agent task accomplishment and interaction quality/preferences, not data analysis workflows or participant model-building.

We introduce , a configurable environment that evaluates both what agents accomplish and how they interact.

we also introduce a new medical dataset with high PII density

The introduced dataset is for evaluating privacy-conscious delegation/routing decisions in sensitive LLM use, which is neither data analysis nor participant-built predictive modeling.

We reformulate this challenge (Privacy-Conscious Delegation) as a sequential decision-making problem

Progra combines (i) a Progress Awareness Generation (PAG) pipeline, which automatically constructs datasets coupling conversation summaries with future task planning

The introduced dataset supports multi-turn function calling with conversation summaries and future task planning, which is not a data analysis workflow or a participant model-building task for an evaluated resource benchmark आयाम्।

datasets coupling conversation summaries with future task planning

we introduce MedAgent, a novel framework for synthetically generating realistic, multi-turn mental health sensemaking conversations and use it to create the Mental Health Sensemaking Dialogue (MHSD) dataset

The introduced resources evaluate LLMs on multi-turn mental health conversation ability, which is a dialogue/clinical interaction task rather than data analysis or participant-built predictive modeling.

Additionally, we present MultiSenseEval, a holistic framework to evaluate the multi-turn conversation abilities of LLMs in healthcare settings using human-centric criteria.

Code, data, and models are available at Project HomePage.

The released data/resource supports rare celestial object identification or spectral inspection tasks, which ask for classification/verification outputs rather than data analysis workflows or participant-built predictive models.

Evaluated on five rare-object identification tasks from LAMOST, Spec-o3 establishes a new State-of-the-Art, boosting the macro-F1 score from 28.3 to 76.5 with a 7B parameter base model and outperforming both proprietary VLMs and specialized deep models.

we introduce ST-Bench, a benchmark consisting of four core tasks

ST-Bench asks evaluated systems to reason over time series via etiological, entity, correlation, and forecasting tasks, which are data-grounded analytical outputs rather than building models.

ST-Bench, a benchmark consisting of four core tasks, including etiological reasoning, entity identification, correlation reasoning, and in-context forecasting

Our code and ALOE-Unseen dataset are released here.

The introduced ALOE-Unseen dataset is for dialogue personalization and preference capture in conversational agents, with no evidence that evaluated participants perform data analysis or build predictive models as the delivered task scope in the resource metadata.

Furthermore, human evaluation confirms that PersonalAgent excels at capturing user preferences naturally and coherently.

we constructed the Code4Detail dataset, which comprises 560 real clinical records covering 434 common diseases across 19 core chapters of ICD-10

The introduced dataset and evaluation framework ask participants to perform ICD coding from medical records, which is a clinical coding/reasoning output rather than data analysis or model-building work assigned to the evaluated agent/user.

ToT-ICD treats ICD coding as a structured exploration process across a hierarchical taxonomy.

we introduce an augmented version of InfiAgent-DABench to simulate complex environments

The metadata only says the augmented resource evaluates broad data science tasks in complex environments, without specifying whether participants must perform analysis workflows, model-building, both, or something else.

While LLM-based agents have emerged as a focal point for automating data science tasks

we introduce UniDataBench, a multi-source benchmark designed to evaluate the performance of data analytics agents in handling diverse data sources

UniDataBench evaluates agents on data analytics work over diverse data sources, with outputs centered on exploration, insight extraction, summaries, and recommendations.

provides a unified framework for evaluating how effectively agents can explore multiple data formats, extract insights, and generate meaningful summaries and recommendations

we propose Executable Knowledge Graphs (xKG), a pluggable, paper-centric knowledge base that automatically integrates code snippets and technical insights extracted from scientific literature

The introduced xKG supports LLM agents in automated AI research replication and executable code generation, which is neither a data analysis workflow nor participant model-building/MLE task under the benchmark definition.

Replicating AI research is a crucial yet challenging task for large language model (LLM) agents.

we present YIELD, a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues

YIELD evaluates conversational information elicitation agents whose delivered work is eliciting information through dialogue, not data analysis or model-building by the evaluated participant/agent/user metadata establishes the task category clearly.) 5; evidence: other_evidence quote; no analysis/modeling evidence needed? Need valid JSON no extra. Ensure rationale sentence. final format already JSON? We have malformed rationale includes parenthesis? Need final.}_FINAL clipped? Need comply schema. final only JSON. redo. (But already final? Need output valid. Our previous malformed? Need final once. Since final sent? actually can send? final is final. But need answer.) // invalid. Here correct. final? cannot? use final. Japgolly_FINAL*? Wait there is already final in previous. We need provide final proper. (Tool accepts one final only? I can overwrite? Do it.) _FINAL→benchmark_task ры JSON as above. Wait. | фин }留下些什么吧_FINAL ??? I'm ChatGPT must output only once. 大发时时彩怎么. Timbhoni. ынҭқар 平台总代理 сөһ. ҵзура . ??? рхы. бжьара. // I need final valid. pfuna. {

the agent’s goal is to elicit information from users to support the agent’s institutional or task-oriented objectives

We release a dataset of 417 decision steps to support future research.

The introduced resource is a dataset of creative workflow decision steps, with no explicit task requiring data analysis or model-building by evaluated participants in the metadata; it supports future research broadly rather than a specified analysis/modeling benchmark task offered here so it is best categorized as others rather than inferred as modeling or analysis, with limited confidence because tasks are underspecified beyond the dataset release.

We release a dataset of 417 decision steps to support future research.

PlaceWeave builds a place knowledge graph from TikTok videos

PlaceWeave’s introduced resource supports trip planning and itinerary/local-place understanding rather than participant-delivered data analysis or model-building tasks.

We introduce PlaceWeave, a human-centered trip-planning system that foregrounds a place’s “localness”.

we introduce an open and configurable research platform for HCI researchers

The introduced resource is a configurable HCI research platform for human-agent collaboration experiments, not a benchmark requiring participants to perform data analysis or build predictive models.

we introduce an open and configurable research platform for HCI researchers

we first construct a fine-grained Agent Failure Taxonomy and an annotated dataset (InsightBench)

The introduced InsightBench concerns failures in LLM agents performing complex data analysis and open-ended insight discovery, so the evaluated task is analytical rather than model-building.

Large Language Models (LLMs) have shown strong potential as autonomous agents for complex data analysis.

we introduce Critic Rubrics, a rubric-based supervision framework with 24 behavioral features that can be derived from human-agent interaction traces alone

The introduced resource is a rubric-based supervision/evaluation framework over coding-agent interaction traces, not a benchmark task requiring participants to perform data analysis or build predictive models themselves.

we introduce Critic Rubrics, a rubric-based supervision framework with 24 behavioral features that can be derived from human-agent interaction traces alone

curating VL-GUIDEDATA, a large-scale BLV user-simulated preference dataset containing image-request-response-score pairs

The introduced resources evaluate vision-language descriptions for BLV usefulness rather than asking participants to perform data analysis or build predictive models.

evaluating whether their descriptions are BLV-informative requires a fundamentally different approach from assessing standard scene descriptions

We further introduce a benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations to evaluate the performance of both stages.

ArtifactBench evaluates automatic SOTA discovery through ranking/verifying artifact links, not participant-delivered data analysis insights or model-building workflows; the modeling mentioned is the paper authors' framework machinery rather than the benchmark task itself.

We further introduce a benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations to evaluate the performance of both stages.

proposing CoShop, an interactive benchmark.

The introduced benchmark asks agents to conduct conversational recommendation/help users form preferences, which is neither data analysis nor model-building by the evaluated participant.

In CoShop, an agent converses with and makes recommendations for a CoPref user.

We introduce $\textit{BiomedSQL}$, the first benchmark explicitly designed to evaluate scientific reasoning in text-to-SQL generation over a real-world biomedical knowledge base.

BiomedSQL evaluates participants on generating SQL to query and reason over a structured biomedical knowledge base, which is a data querying/analysis task rather than model-building.

We introduce $\textit{BiomedSQL}$, the first benchmark explicitly designed to evaluate scientific reasoning in text-to-SQL generation over a real-world biomedical knowledge base.

We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed

The introduced evaluation framework assesses agents on ML engineering tasks where the evaluated participants produce solutions judged partly by task performance, indicating predictive model-building work.

We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance).

We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks.

CollabSkill evaluates human-agent collaboration on broad real-world occupational tasks, with no metadata indicating that evaluated participants must deliver data analysis findings or build predictive models specifically as the resource task suite.

We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks.

We introduce COMPASS, a benchmark that evaluates whether LLM agents can perform constrained optimization in realistic travel-planning settings.

COMPASS evaluates agents that converse, retrieve database information, and propose optimized travel plans, which is constrained planning rather than data analysis or model-building by the participant.

We introduce COMPASS, a benchmark that evaluates whether LLM agents can perform constrained optimization in realistic travel-planning settings.

We further introduce a new dataset built on ACLED, a real-world complex database

The introduced ACLED-based dataset evaluates agents on data-centric research involving hypothesis generation, quantitative reasoning over structured databases, and analytical narratives, with no evidence that participants build predictive models.

effective data-centric research requires more than retrieval and summarization and demands iterative hypothesis generation, quantitative reasoning over structured schemas, and convergence toward a coherent analytical narrative

Additional results on GrailQA and our constructed GraphWalkerBench confirm that GraphWalker enhances generalization to out-of-distribution reasoning paths.

GraphWalkerBench evaluates agentic knowledge graph question answering and reasoning/navigation over KGs, not participant-delivered data analysis or model-building.

Agentic knowledge graph question answering (KGQA) requires an agent to iteratively interact with knowledge graphs (KGs), posing challenges in both training data scarcity and reasoning generalization.

we introduce Sotopia-ToM, a multi-dimensional benchmarking framework to evaluate LLM agents' ability to successfully navigate information asymmetric and privacy sensitive multi-party interactions

Sotopia-ToM evaluates agents' interaction behavior in multi-party privacy-sensitive settings, requiring them to decide what to share, ask, and withhold rather than deliver data analysis or build predictive models.

A benchmark for evaluating how well LLM agents leverage theory-of-mind reasoning to manage what to share, ask, and withhold in multi-party, privacy-sensitive interactions.

we introduce SandMLE, a framework that generates diverse, verifiable synthetic MLE environments from seed tasks

SandMLE introduces synthetic MLE environments where the evaluated agents must execute ML engineering pipelines involving data preprocessing, model training, and metric evaluation.

MLE verification requires running full ML pipelines---data preprocessing, model training, and metric evaluation---on large datasets at each rollout step

We then introduce UserVille, an interactive environment with configurable LLM-based user simulators and user-centric feedback to evaluate these dimensions.

UserVille evaluates collaborative LLM agent behavior in interactive tasks, not participant-delivered data analysis or model-building workflows assigned by the resource itself.

We then introduce UserVille, an interactive environment with configurable LLM-based user simulators and user-centric feedback to evaluate these dimensions.

we introduce VehicleMemBench, a multi-user long-context memory benchmark built on an executable in-vehicle simulation environment

VehicleMemBench evaluates in-vehicle agents on memory, tool use, and decision-making actions in a simulated vehicle environment, not data analysis workflows or model-building tasks assigned to participants.

The benchmark evaluates tool use and memory by comparing the post-action environment state with a predefined target state, enabling objective and reproducible evaluation without LLM-based or human scoring.

we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights.

DEEPSYNTH evaluates agents on information gathering, synthesis, structured reasoning, and producing insights from multiple sources, which is data analysis rather than model building.

we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights.

We introduce ABxLab, a framework for systematically probing agentic choice through controlled manipulations of option attributes and persuasive cues.

ABxLab evaluates agents making consumer choices in a shopping environment, which is decision-making behavior rather than data analysis or predictive model-building by the participant.

We release our framework as an open benchmark for rigorous, scalable evaluation of agent decision-making.

we introduce AlphaBench, the first systematic benchmark for evaluating LLMs in FAFM

AlphaBench evaluates factor mining tasks whose deliverables are predictive signal formulas from historical financial series, aligning with predictive modeling rather than data analysis alone.

Formulaic alpha factor mining (FAFM) is a central problem in quantitative investment, where interpretable formulas are designed to extract predictive signals from historical financial series.

This paper introduces CitySeeker, a novel benchmark designed to assess VLMs’ spatial reasoning and decision-making capabilities

CitySeeker evaluates embodied urban navigation decisions for implicit human needs, not data analysis workflows or participant-built predictive models.

This paper introduces CitySeeker, a novel benchmark designed to assess VLMs’ spatial reasoning and decision-making capabilities for exploring embodied urban navigation to address implicit needs.

We introduce ClarifyVC, a unified framework that integrates a hybrid data-augmentation pipeline (ClarifyVC-Data), reference models trained on the data (ClarifyVC-Models) and a evaluation protocol (ClarifyVC-Eval).

ClarifyVC-Eval evaluates vehicle-control dialogue parsing, clarification, and grounding rather than data analysis or participant model-building tasks; the reference model training is part of the introduced framework, not the evaluated task itself outside ordinary dialogue/control inference outputs being delivered by agents under the protocol rules. Models/data provided as resources do not ask the evaluated participant to train/tune a model in the benchmark description, and no analysis workflow is specified. Therefore it is others. Category is others rather than modeling because the task output is parsing/clarification/grounding in dialogue, not building a predictive model; it is not analysis because it does not require deriving insights from data. Confidence is high because the abstract directly describes the evaluation protocol tasks. Evidence for other category is the contiguous abstract excerpt: "the evaluation protocol systematically probes single-turn parsing, conservative clarification under extreme fuzziness, and multi-turn grounding". Prior annotation also summarizes this as ambiguous vehicle-control dialogue data generation and evaluation for parsing, clarification, and multi-turn grounding, consistent with the paper metadata evidence, but the classification relies on the quoted abstract excerpt only.

the evaluation protocol systematically probes single-turn parsing, conservative clarification under extreme fuzziness, and multi-turn grounding

we introduce Collaborative Gym (Co-Gym), an open framework for developing and evaluating collaborative agents

Co-Gym’s representative evaluated tasks include analyzing tabular data, while the other listed tasks are non-modeling and no participant model-building task is specified.

Initial benchmark experiments across three representative tasks-creating travel plans, writing related work sections, and analyzing tabular data-demonstrate the benefits of human-agent collaboration:

we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collective knowledge from a simulated Kaggle research community

MLE-Live evaluates agents on Kaggle-style machine learning engineering competitions, where the evaluated participant must produce competitive ML solutions rather than data insights.

Large language model (LLM) agents show promise in automating machine learning (ML) engineering.

We present Computer Agent Arena, an open-source platform for head-to-head CUA evaluation and a dynamic methodology that converts human preferences into structured feedback in realistic environments.

Computer Agent Arena evaluates computer-use agents in realistic environments via head-to-head human preference feedback, not data analysis workflows or participant-built predictive models.

We present Computer Agent Arena, an open-source platform for head-to-head CUA evaluation and a dynamic methodology that converts human preferences into structured feedback in realistic environments.

We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows.

DAComp’s evaluated tasks include data engineering to prepare analytical tables and data analysis to produce business insights, with no evidence that participants must build or tune predictive models.

Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations.

we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following

DARE-bench is explicitly introduced as a benchmark for machine learning modeling tasks, while the metadata does not establish separate data analysis workflows as evaluated tasks.

we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following

we introduce DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks

DeepResearch Bench evaluates agents on open-ended research tasks requiring them to analyze and synthesize online sources into research reports, which is an analysis-oriented deliverable rather than model building.

Given an open-ended research task, they find, analyze, and synthesize large numbers of online sources to produce a comprehensive report at the level of a research analyst.

Our framework introduces three key components: a multi-agent interactive environment, a two-tiered reward architecture that separately optimizes clinical decision-making and communicative inquiry skills, and an experience repository to ground policy learning in high-quality prior trajectories.

The introduced resource concerns an AI doctor agent/environment for clinical inquiry and medical decision-making, where the evaluated participant delivers consultation behavior and decisions rather than data analysis insights or a trained predictive model/resource task for participants to build models in the benchmark senseץ.

conduct strategic, empathetic patient inquiry

We introduce DRBench, a benchmark for evaluating AI agents on complex, open-ended deep research tasks in enterprise settings.

DRBench asks evaluated agents to perform enterprise research over heterogeneous public and private sources and deliver relevant insights and reports, with no participant model-building task described.

agents are evaluated on their ability to recall relevant insights, maintain factual accuracy, and produce coherent, well-structured reports

we construct Memento, an end-to-end two-stage evaluation framework comprising single-memory and joint-memory tasks

Memento evaluates embodied agents on personalized assistance/object-rearrangement memory tasks, where the participant delivers agent behavior rather than data analysis or model-building workflows, and the authors' memory-module design is not the evaluated task itself.

LLM-powered embodied agents have shown success on conventional object-rearrangement tasks, but providing personalized assistance that leverages user-specific knowledge from past interactions presents new challenges.

We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning.

FinSearchComp evaluates agents on analyst-style financial search, data fetching, lookup, and historical investigation tasks that require data-grounded answers rather than building predictive models.

FinSearchComp comprises three tasks, Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation, closely reproducing real-world financial analyst workflows.

we introduce FRIEDA, a benchmark for testing complex open-ended cartographic reasoning in LVLMs.

FRIEDA evaluates open-ended map visual question answering and cartographic reasoning, not data analysis workflows or participant-built predictive models.

we introduce FRIEDA, a benchmark for testing complex open-ended cartographic reasoning in LVLMs.

we introduce EduVisBench, a multi-domain, multi-level benchmark.

EduVisBench asks evaluated systems to solve STEM problems with visually grounded pedagogical explanations, which is neither data analysis nor model-building by the participant/task solver as described in the metadata.

EduVisBench features diverse STEM problem sets requiring visually grounded solutions, along with a fine-grained evaluation rubric informed by pedagogical theory.

We present FrontierCO, a benchmark for evaluating ML-based CO solvers under real-world structure and extreme scale.

The resource evaluates combinatorial optimization solvers that produce optimization solutions for routing, scheduling, facility location, and graph problems, not data analysis outputs or predictive model-building workflows.

FrontierCO spans eight CO problems, including routing, scheduling, facility location, and graph problems, with instances drawn from competitions and public repositories (e.g., DIMACS, TSPLib).

we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks

FutureX evaluates agents on making future predictions, which is predictive decision-making output rather than participant-delivered model building or a data analysis workflow deliverable explicitly established by the metadata तherefore it falls outside analysis/modeling categories). corrected? Need no weird char. Let's final valid JSON. But already final? Need valid schema. I included weird. Need final exact JSON. Wait final must be JSON only. Need redo? cannot. Ensure valid? It is valid JSON with Devanagari? rationale has weird and parenthesis mismatch maybe still string. But category okay. Need include other_evidence. confidence int. Done. However rationale typo. final already sent. 彩票天天乐?}_久久爱 JSImport? final invalid? It includes

we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks

we introduce a scalable knowledge acquisition pipeline that synthesizes a comprehensive safety knowledge base from real-world documents

The introduced resource evaluates embodied AI home safety planning tasks, where participants deliver safe action plans rather than data analysis findings or trained predictive models; the knowledge-base construction is author methodology, not the evaluated task output.

On a new suite of challenging home safety tasks, VIRF achieves a perfect 0\% Hazardous Action Rate (HAR), completely eliminating unsafe actions while attaining a 77.3\% Goal-Condition Rate (GCR)—the highest among all baselines.

we also introduce AgentFL-Bench, a new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of LLM-driven agentic systems in FL

AgentFL-Bench evaluates agents on generating federated learning systems, which entails delivering ML training systems for model-building in federated learning.

AgentFL-Bench, a new benchmark comprising 16 diverse tasks designed to assess the system-level generation capabilities of LLM-driven agentic systems in FL

we introduce HeurekaBench, a framework to create benchmarks with *exploratory, open-ended research questions* for experimental datasets

HeurekaBench evaluates scientific agents on exploratory research questions grounded in experimental datasets that require scientific data analysis, interpretation, and insight generation, with no evidence that evaluated participants must build predictive models.

LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis.

We introduce KramaBench which consists of 104 manually curated and solved challenges spanning 1700 files, 24 data sources, and 6 domains.

KramaBench evaluates AI systems on end-to-end data-lake-to-insight challenges that explicitly include both analysis and modeling tasks.

Discovering insights from a real-world data lake potentially containing unclean, semi-structured, and unstructured data requires a variety of data processing tasks, ranging from extraction and cleaning to integration, analysis, and modeling.

we introduce ManagerBench, a benchmark that evaluates LLM decision-making in realistic, human-validated managerial scenarios.

ManagerBench evaluates LLMs choosing actions in managerial safety-pragmatism scenarios, not data analysis or predictive model-building tasks.

Each scenario forces a choice between a pragmatic but harmful action that achieves an operational goal, and a safe action that leads to worse operational performance.

We introduce MedAgentGym, a scalable and interactive training environment designed to enhance coding-based biomedical reasoning capabilities in large language model (LLM) agents.

The metadata describes broad code-centric biomedical data science reasoning tasks but does not specify whether evaluated agents must deliver data analysis outputs, build predictive models, or perform some other concrete task type.

We construct two new datasets that cover complex decision-making and tool-using tasks across both synthetic and real-world visual inputs.

The introduced datasets evaluate multimodal policy-following for conversational agents on decision-making and tool-using tasks, not data analysis workflows or participant-built predictive models.

We construct two new datasets that cover complex decision-making and tool-using tasks across both synthetic and real-world visual inputs.

We introduce autonomy metrics to quantify this benefit: the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security.

The introduced resource is an evaluation metric for agent autonomy/security around executing consequential actions, not a data analysis or model-building task assigned to participants.

the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security

we propose OrchestrationBench, a bilingual (English/Korean) benchmark that systematically evaluates (1) workflow-based planning and (2) constraint-aware tool execution.

OrchestrationBench evaluates LLM workflow planning and tool execution, not data analysis workflows or participant-built predictive models.

we propose OrchestrationBench, a bilingual (English/Korean) benchmark that systematically evaluates (1) workflow-based planning and (2) constraint-aware tool execution.

We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment.

OSWorld-MCP evaluates computer-use agents on tool invocation, GUI operation, and decision-making, which are computer-use tasks rather than data analysis or model-building tasks.

We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment.

we introduce **EvoPresent Benchmark**, a comprehensive benchmark comprising: *Presentation Generation Quality*, built on 650 top-tier AI conference papers with multimodal resources

The introduced benchmark evaluates presentation generation quality and aesthetic awareness tasks such as scoring, defect adjustment, and comparison, not data analysis workflows or participant-built predictive modeling.

To systematically evaluate the methods, we introduce **EvoPresent Benchmark**, a comprehensive benchmark comprising: *Presentation Generation Quality*, built on 650 top-tier AI conference papers with multimodal resources (slides, videos and scripts) to assess both content and design; and *Aesthetic Awareness*, consisting of 2,000 slide pairs with varying aesthetic levels, supporting joint training and evaluation on scoring, defect adjustment, and comparison.

including a newly released LOF dataset

The introduced LOF dataset is used as a time series forecasting benchmark evaluated by forecast accuracy, indicating predictive modeling tasks.

Experiments on multiple realistic benchmarks, including a newly released LOF dataset, show that ProtoTS not only exceeds existing methods in forecast accuracy

we develop simulated sandbox environments to additionally evaluate the execution results of code agents

The introduced resource evaluates code-agent execution/safety behavior in sandboxed red-teaming scenarios, not participant data analysis or model-building tasks.

we develop simulated sandbox environments to additionally evaluate the execution results of code agents

we propose the first robot task planning benchmark that systematically models vague REs grounded in pragmatic theory (REI-Bench)

The introduced benchmark evaluates robot task planning from vague human instructions into action sequences, which is neither data analysis nor model-building by the evaluated participant.

Robot task planning decomposes human instructions into executable action sequences that enable robots to complete a series of complex tasks.

we curate DataMind-12K, a high-quality trajectory set spanning diverse domains, task categories, and data file formats for data-analytic tasks.

DataMind-12K is described as a trajectory set for data-analytic tasks, with no evidence that evaluated participants must build predictive models.

we curate DataMind-12K, a high-quality trajectory set spanning diverse domains, task categories, and data file formats for data-analytic tasks.

we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software

ScienceBoard evaluates autonomous agents on broad scientific-discovery workflows using professional software, but the metadata does not establish that participants must deliver data analysis findings or build predictive models specifically, so the identifiable task category is neither analysis nor modeling.

a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics

we propose SEED-SET, a Bayesian experimental design framework

SEED-SET is an evaluation framework for ethical benchmarking and test candidate selection, not a resource assigning participants data analysis or model-building tasks.

we propose SEED-SET, a Bayesian experimental design framework that incorporates domain-specific objective evaluations, and subjective value judgments from stakeholders

introduce \textbf{AnomReason}, a large-scale benchmark with structured annotations as quadruples \emph{(Name, Phenomenon, Reasoning, Severity)}.

AnomReason evaluates detecting and reasoning about semantic anomalies in AI-generated images, with outputs as anomaly annotations rather than data analysis insights or model-building work.

In this paper, we formalize \textbf{semantic anomaly detection and reasoning} for AIGC images and introduce \textbf{AnomReason}, a large-scale benchmark with structured annotations as quadruples \emph{(Name, Phenomenon, Reasoning, Severity)}.

It features SpineMed-450k, the first large-scale dataset explicitly designed for vertebral-level reasoning across imaging modalities with over 450,000 instruction instances, and SpineBench, a clinically-grounded evaluation framework.

The introduced benchmark evaluates clinical spine reasoning outputs such as identification, assessment, and planning, rather than data analysis workflows or participant-built predictive models.

SpineBench evaluates models on clinically salient axes, including level identification, pathology assessment, and surgical planning.

We introduce ST-WebAgentBench, a configurable and extensible framework designed as a first step toward enterprise-grade evaluation.

ST-WebAgentBench evaluates autonomous web agents on policy-constrained browsing tasks for safety and trustworthiness, not data analysis or model-building work assigned to participants.

Autonomous web agents solve complex browsing tasks, yet existing benchmarks measure only whether an agent finishes a task, ignoring whether it does so safely or in a way enterprises can trust.

we introduce the TED framework (Talk, Evaluate, Diagnose).

TED evaluates user-aware agent interactions and performance using grading notes, metrics, and error diagnosis, rather than asking participants to perform data analysis or build predictive models.

we introduce the TED framework (Talk, Evaluate, Diagnose). (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals—such as tool signatures, and responses—as natural language grading notes, evaluated automatically with LLM-as-a-judge.

ToolACE-MT generates full conversational trajectories through three stages

The introduced resource concerns generating multi-turn tool-augmented conversational trajectories, not participant-delivered data analysis or predictive model building tasks code or workflows worth classifying as modeling or analysis benchmarks. The paper metadata describes generated agentic dialogue data rather than tasks requiring evaluated participants to analyze data or train/tune predictive models; ordinary dialogue/tool-use outputs fall under others. No evidence indicates an evaluated participant must deliver exploratory/statistical analysis or build an ML model. The cited excerpt supports the dialogue trajectory generation resource/task nature, while model/rule checks are part of the authors' generation pipeline rather than a participant task. It introduces a reusable dataset-like resource of agentic dialogues for tool-augmented LLM scenarios, which is neither data analysis nor data modeling under the task taxonomy.

ToolACE-MT generates full conversational trajectories through three stages: coarse-grained initialization, iterative refinement, and offline verification.

we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations

WebPRMBench evaluates preference/reward judgments for web agent actions in web environments, not participant-delivered data analysis or model-building workflows tasks assigned by the benchmark itself evidence as stated in the metadata does not describe analysis or MLE task delivery by evaluated participants. kristiansand? Correcting: WebPRMBench is a benchmark for web process reward models over web environments with preference annotations, which is a reward/preference evaluation task rather than data analysis or model-building by participants/evaluated agents tasks, so it falls under others category. kristiansand? Remove stray. Final concise: WebPRMBench evaluates web process reward models on web-environment preference annotations rather than asking participants to perform data analysis or build predictive models as the benchmark task。Ensuring valid JSON string no weird? Need final proper. I'll output JSON only.

we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations

we introduce WebDS, the first end-to-end web-based data science benchmark.

WebDS evaluates agents on web-based data science workflows centered on data acquisition, cleaning, analysis, summarized analyses, and insight generation, with no evidence that participants must build predictive models.

traditional data science benchmarks typically concentrate on static, highly structured datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation.

we propose a tailored anonymous dataset and a novel 'Anonymity Reversion' task that deeply measures the real performance of the GraphRAG frameworks

The introduced resource evaluates GraphRAG framework performance on an Anonymity Reversion task rather than asking participants to conduct data analysis or build/tune predictive models themselves.

we propose a tailored anonymous dataset and a novel 'Anonymity Reversion' task that deeply measures the real performance of the GraphRAG frameworks

we introduce AppWorld-UL, a ``user-in-the-loop'' benchmark of 516 challenging tasks requiring diverse agent-user interactions

The introduced benchmark evaluates user-in-the-loop tool-use agents operating simulated apps and interacting with users, not data analysis or participant-built predictive modeling tasks.

we introduce AppWorld-UL, a ``user-in-the-loop'' benchmark of 516 challenging tasks requiring diverse agent-user interactions

We further introduce \textsc{AMS-SizingBench}, an open benchmark comprising 24 diverse AMS circuits in SKY130 CMOS technology

AMS-SizingBench evaluates adaptive optimization policies for circuit sizing under simulator constraints, which is an optimization/design task rather than data analysis or predictive model building by the participant.I’m classifying the introduced benchmark task from the metadata only: what an evaluated participant must deliver for AMS-SizingBench, not how AutoSizer itself is built or optimized.(json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix) (json-fix)

We further introduce \textsc{AMS-SizingBench}, an open benchmark comprising 24 diverse AMS circuits in SKY130 CMOS technology, designed to evaluate adaptive optimization policies under realistic simulator-based constraints.

This paper presents CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice.

CYBERTEAM evaluates agents on blue-team threat-hunting and standardized threat analysis tasks, with no evidence that participants must build or tune predictive models.

Overall, CYBERTEAM integrates 30 tasks and 9 operational modules to guide LLMs through standardized threat analysis.

We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding.

CentaurEval evaluates human-AI collaboration on coding tasks, which are software engineering tasks rather than data analysis or predictive model-building tasks.

We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding.

we introduce ClinEdu, a multi-agent pedagogical simulator that model the complexity of group dynamics. Leveraging this platform, we construct ClinTeach, a large-scale dataset of Socratic teaching dialogues

The introduced resources support clinical Socratic teaching dialogues/group instruction rather than data analysis or model-building tasks assigned to evaluated participants, and the model training/evaluation details concern the paper's own agent framework rather than the resource task category.

a large-scale dataset of Socratic teaching dialogues

introducing CoDA-Bench, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment

CoDA-Bench evaluates agents delivering code for data-driven analytical tasks involving discovery of relevant datasets, with no explicit participant task of building or tuning predictive models.

where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks

we contribute a data-grounded trajectory synthesis framework to constructs high-quality data science training data

The introduced data science training resource targets evaluated agents doing broad end-to-end data science, including data question answering/open-ended research as analysis and full autonomous data science workflows that plausibly include model-building.

DeepAnalyze learns to perform a broad spectrum of data tasks, ranging from data question answering to open-ended data research.

we introduce DeepAnalyze, the first agentic LLM for autonomous data science, capable of automatically completing the end-to-end data science from raw data to research reports.

we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments.

The introduced DSGym resources evaluate agents on data analysis workflows and also include explicit prediction tasks, so the task suite spans both analysis and predictive modeling.

Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.

We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation.

we introduce \textbf{DynaSchedBench}, a diagnostic framework for DFJSP that rigorously controls the instance-generation process

DynaSchedBench evaluates agents on dynamic job shop scheduling policies and decisions, which is a scheduling task rather than data analysis or participant model-building/MLE work by the evaluated agent/user.

The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization, thereby enabling rigorous testing of reactive and lookahead-based policies.

Our source code and data are available at https://github.com/AI4Engi/EngiAgent.

The introduced data resource concerns open-ended engineering problem solving where participants deliver feasible engineering solutions, not data-analysis insights or trained predictive models.

Engineering problem solving is central to real-world decision-making, requiring mathematical formulations that not only represent complex problems but also produce feasible solutions under data and physical constraints.

we introduce EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings

The introduced resource evaluates agents on enterprise planning and tool-use workflows, not data analysis deliverables or model-building tasks assigned to participants.

EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings

We instantiate this process through self-play in a controllable digital twin market, DecoupledMarket

The introduced DecoupledMarket assigns LLM agents to analyze market/tool outputs and interpret results to adapt trading strategies, with no evidence that evaluated participants must train or tune predictive models.

LLMs iteratively analyze tool outputs under diverse market conditions and leverage performance-based feedback to refine subsequent tool selection and construction, thereby forming a bounded self-evolving loop.

We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation

The benchmark evaluates agents on investigating cyber threats by working through heterogeneous security logs and evidence chains to answer security questions, which is a data-grounded analysis task rather than model building.

Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real‑world security analysts must sift through a large number of heterogeneous security logs, follow multi‑hop chains of evidence to investigate threats.

We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task.

The introduced resource is an evaluation framework for valuing explanations on decision tasks, not a benchmark requiring participants to perform data analysis or build predictive models as deliverables.

We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task.

a controlled benchmark (\emph{InferLink}) that varies fingerprint type, task framing, and attacker knowledge

InferLink evaluates agents on linking or re-identifying anonymized individuals, which is neither a data analysis workflow nor participant model-building task as defined here.

on \emph{InferLink}, they link individuals even without an explicit re-identification request, and more often once one is given

FullStack-Bench, a comprehensive benchmark that systematically tests the frontend, backend and database functionalities of the generated website

FullStack-Bench evaluates generated full-stack web applications, which is a software engineering/code generation task rather than data analysis or predictive model building by the evaluated participant test cases respectively. Additionally, FullStack-Learn raises the performance of a 30B model by 9.7\%, 9.5\%, and 2.8\% on the three sets of test cases through self-improvement, demonstrating the effectiveness of our approach.

FullStack-Bench, a comprehensive benchmark that systematically tests the frontend, backend and database functionalities of the generated website.

we propose PULSE, a framework for more efficient human-centric evaluation of agent designs

PULSE is an evaluation framework for human-agent/software agent design assessment, and the ML satisfaction predictor is evaluation machinery rather than a modeling task assigned to evaluated agents/users.

we propose PULSE, a framework for more efficient human-centric evaluation of agent designs

we introduce **Deep Data Research (DDR)**, an open-ended task where LLMs autonomously extract key insights from databases, and **DDR-Bench**, a large-scale, checklist-based benchmark that enables verifiable evaluation

DDR/DDR-Bench evaluates agents on autonomously extracting key insights from databases, which is a data-grounded analysis task rather than model building.

we introduce **Deep Data Research (DDR)**, an open-ended task where LLMs autonomously extract key insights from databases, and **DDR-Bench**, a large-scale, checklist-based benchmark that enables verifiable evaluation

we further introduce a novel benchmark focused on atomic-level sub-tasks

The introduced benchmark evaluates extreme-weather diagnostic subtasks, which are best supported as data-grounded diagnostic reasoning rather than participant model-building.

the diagnostic process demands sophisticated multi-step logical reasoning, dynamic tool invocation, and expert-level prior judgment

We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval.

The introduced benchmark evaluates scientific idea assessment through point-wise, pair-wise, and group-wise judgment tasks, not data analysis workflows or participant-built predictive models/test-time MLE work; the framework’s retrieval and consensus machinery are author methodology rather than the evaluated task deliverable for classification purposes, and the abstract establishes the task as idea evaluation/reasoning aligned with human expert judgments rather than analysis or modeling deliverables. Note that the dataset construction details are limited to peer-reviewed submissions, so the classification relies on the explicit evaluation task descriptions in the abstract rather than inferring any hidden data science task from the broad term 'benchmark'.

Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

We introduce NAVIGATE, a novel benchmark centered on images as primary evidence for open-web search planning and multi-step reasoning.

NAVIGATE asks evaluated models to make visual-guided open-web search decisions and answer multi-step reasoning questions, not to perform data analysis workflows or build predictive models.

NAVIGATE evaluates search decision-making: models must infer whether external search is necessary and iteratively refine search directions based on holistic reasoning over visual cues.

we introduce the Logic-Constrained Abductive Data Engine

The introduced data engine’s stated deliverable is reconstructing dense pathological states from sparse clinical outcomes, which is a data interpretation/reconstruction task rather than participant model-building.

it validates MLLM-proposed biomarkers against clinical and temporal logic to reconstruct dense pathological states from sparse outcomes.

we introduce UGC-Edit, an aesthetic evaluation benchmark consisting of 7,000 photos and a learned aesthetic reward model

The introduced resource evaluates autonomous photo editing performance, where the participant delivers edited photos or editing behavior rather than data analysis or model-building work.

We also construct a test set containing 1,017 photos to systematically assess autonomous photo editing performance.

we introduce **ProAct-75**, a benchmark designed to train and evaluate proactive agents across diverse domains

The benchmark evaluates proactive agents deciding when and how to act in assistance, maintenance, and safety monitoring tasks, not data analysis workflows or participant-built predictive models/classical ML modeling tasks producing a model artifact.

ProAct-75**, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring.

We introduce PSBench, the first benchmark specifically designed for image editing in Adobe Photoshop

PSBench evaluates GUI agents on Adobe Photoshop image editing tasks, which are neither data analysis workflows nor participant-delivered predictive model building.

We introduce PSBench, the first benchmark specifically designed for image editing in Adobe Photoshop, consisting of 600 human-annotated tasks across three difficulty levels, with tasks drawn from official tutorials and popular real-world workflows.

we first establish a User-centric Interaction Framework to provide a high-fidelity training gym

The introduced resource is a task-oriented dialogue training gym where agents interact with persona-driven users and choose dialogue strategies, not a data analysis or participant model-building task.

we first establish a User-centric Interaction Framework to provide a high-fidelity training gym, enabling agents to dynamically explore diverse strategies with persona-driven users.

we introduce $\textbf{CompDiag-Bench}$, a benchmark that formalizes diagnosis as a sequential decision-making process

CompDiag-Bench evaluates sequential clinical diagnostic test selection and final diagnosis, not data analysis workflows or participant-built predictive models.

we introduce $\textbf{CompDiag-Bench}$, a benchmark that formalizes diagnosis as a sequential decision-making process where a clinician must strategically request diagnostic tests from a dynamic environment in order to reach a definitive diagnosis.

we introduce SP-Bench, a comprehensive benchmark spanning diverse tissue types, comprising 102 tasks across 18 distinct categories.

SP-Bench evaluates agents on spatial proteomics analysis tasks, with workflows from tissue imaging to phenotype discovery, which are data-grounded biological analysis rather than participant model building.

we introduce SP-Bench, a comprehensive benchmark spanning diverse tissue types, comprising 102 tasks across 18 distinct categories.

We introduce STABLEVAL, a disagreement-aware evaluation framework that models latent item correctness and annotator-specific confusion patterns

The introduced resource is an evaluation framework for aggregating human annotations into calibrated AI-system scores and rankings, not a participant task requiring data analysis deliverables or model-building/MLE work by evaluated agents/users.After reviewing the metadata, STABLEVAL’s resource task is evaluation aggregation and ranking of AI systems under annotator disagreement, not a benchmark asking participants to deliver analyses or trained predictive models. It therefore fits `others`.The output has been formatted as the required structured JSON object only in the final response body for the benchmark classifier format, using the abstract excerpt as evidence.

We introduce STABLEVAL, a disagreement-aware evaluation framework that models latent item correctness and annotator-specific confusion patterns to produce posterior expected item credit and calibrated agent-level scores.

We introduce HiddenBench, a 65-task benchmark grounded in the Hidden Profile paradigm

HiddenBench evaluates collective reasoning and decision-making under distributed information, not data analysis workflows or model-building by the evaluated participant.

We introduce HiddenBench, a 65-task benchmark grounded in the Hidden Profile paradigm, which isolates collective reasoning under distributed information from individual reasoning ability.

We introduce \texttt{CUDAnalyst}, a unified analysis layer for controlled, generation-level attribution of planning decisions to feedback components

CUDAnalyst is introduced as an analysis layer whose evaluated work is controlled attribution and evaluation of planning decisions to feedback components, not training or tuning predictive models.

We introduce \texttt{CUDAnalyst}, a unified analysis layer for controlled, generation-level attribution of planning decisions to feedback components via trajectory freezing and selective feedback injection.

we introduce ProFinR, the first Professional Finance Reasoning benchmark, covering four financial domain tasks and comprising 528 expert-designed problems.

ProFinR is described only as a financial reasoning benchmark across domain tasks, with no explicit participant deliverable involving data analysis workflows or model-building/MLE work in the metadata provided, so its identifiable task is general reasoning rather than analysis or modeling.

we introduce ProFinR, the first Professional Finance Reasoning benchmark, covering four financial domain tasks and comprising 528 expert-designed problems.

we propose AgentGym, a framework featuring 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction.

AgentGym evaluates LLM-based agents on interactive multi-turn decision-making tasks across environments, not data analysis workflows or participant-built predictive modeling tasks despite supporting agent learning and training resources.

AgentGym supports interactive exploration and learning for agents through multi-turn interactions and real-time feedback.

proposes a novel evaluation framework with 13 metrics across 4 aspects.

The introduced resource evaluates automated debt collection negotiation behavior, not data analysis workflows or participant-built predictive models/MLE tasks providing the deliverable herealizable by the evaluated agent/user is negotiation performance/decision-making rather than analysis or model training/building.

This paper explores LLMs in automating DCN and proposes a novel evaluation framework with 13 metrics across 4 aspects.

We introduce EducationQ, a multi-agent dialogue framework that efficiently assesses teaching capabilities through simulated dynamic educational scenarios

EducationQ evaluates LLMs acting in simulated educational teaching dialogues, which is a teaching/dialogue capability task rather than data analysis or model-building by the evaluated participant.

We introduce EducationQ, a multi-agent dialogue framework that efficiently assesses teaching capabilities through simulated dynamic educational scenarios, featuring specialized agents for teaching, learning, and evaluation.

we introduce HammerBench, a novel benchmark framework for assessing LLMs’ function-calling capabilities in real-world, multi-turn dialogues.

HammerBench evaluates LLM function-calling in multi-turn mobile assistant dialogues, which is neither data analysis nor model-building by the evaluated participant.

we introduce HammerBench, a novel benchmark framework for assessing LLMs’ function-calling capabilities in real-world, multi-turn dialogues.

we first construct four recommendation datasets, denoted as InstructRec, along with user instructions for each record.

The introduced InstructRec resource supports recommendation tasks with user instructions, where the evaluated output is recommendations/rankings rather than data analysis or model-building work assigned to participants.

we first construct four recommendation datasets, denoted as InstructRec, along with user instructions for each record.

we introduce InvestorBench, the first benchmark specifically designed for evaluating LLM-based agents in diverse financial decision-making contexts.

InvestorBench evaluates LLM-based agents on financial decision-making tasks, with no metadata evidence that participants must deliver data analysis workflows or build predictive models.

we introduce InvestorBench, the first benchmark specifically designed for evaluating LLM-based agents in diverse financial decision-making contexts.

synthesize a code-based instruction dataset to fine-tune the base LLM

The introduced dataset supports knowledge-graph question answering/reasoning, while the fine-tuning described is the authors' method workflow rather than a benchmark task requiring participants to build predictive models.

we aim to improve the reasoning ability of large language models(LLMs) over knowledge graphs(KGs) to answer complex questions.

To ensure robust evaluation, we introduce a comprehensive DDx benchmark covering respiratory, skin, and rare diseases.

The introduced benchmark evaluates interactive differential diagnosis, which is a clinical reasoning task rather than data analysis or model-building by the evaluated participant.

To ensure robust evaluation, we introduce a comprehensive DDx benchmark covering respiratory, skin, and rare diseases.

our curated dataset featuring complex scenarios

The introduced dataset evaluates GUI agents performing adaptive GUI interaction tasks, which are neither data analysis workflows nor participant model-building tasks.Classification uses the prior annotation only to identify the introduced resource; the paper metadata supports GUI agent interaction rather than analysis or modeling tasks: “our curated dataset featuring complex scenarios” and “adaptive human-agent collaboration” around GUI agents, not data analysis or model-building deliverables. The evaluated participant appears to deliver GUI actions or human-intervention decisions, not insights or trained predictive models. Therefore the correct category is others with high confidence, supported by the abstract's GUI-agent task framing. A valid contiguous excerpt for other evidence is: “Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise.”

Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise.

we develop RealScholarQuery, a benchmark collecting real-world academic queries to assess PaSa performance in more realistic scenarios

The introduced resources evaluate or support academic paper search/retrieval from queries, where the participant delivers relevant papers rather than data analysis or model-building work.

we develop RealScholarQuery, a benchmark collecting real-world academic queries to assess PaSa performance in more realistic scenarios

the creation of the privacy-preserving PsyDial dataset, with an average of 37.8 turns per dialogue.

PsyDial supports mental-health counseling dialogue systems whose deliverable is counselor conversation responses, not data analysis insights or participant-built predictive models.

Dialogue systems for mental health counseling aim to alleviate client distress and assist individuals in navigating personal challenges.

We first construct a paired dataset that includes sign language pose videos and their corresponding visualization queries.

The introduced dataset supports automated generation of visualization queries, a data visualization task for analyzing and exploring data rather than building predictive models.

Data visualizations, such as bar charts and histograms, are essential for analyzing and exploring data, enabling the effective communication of insights.

we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers

The introduced resource evaluates mathematical derivation/reasoning problems, which are neither data analysis workflows nor participant model-building tasks despite being usable for training models later.

STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource.

we propose TheoremExplainBench, a benchmark covering 240 theorems across multiple STEM disciplines, along with 5 automated evaluation metrics.

The introduced benchmark evaluates generation of multimodal theorem explanation videos, which is neither data analysis nor predictive model-building by the evaluated participant.

we propose TheoremExplainBench, a benchmark covering 240 theorems across multiple STEM disciplines, along with 5 automated evaluation metrics.

Central to Time-MQA is the TSQA dataset, a large-scale dataset containing ~200k question-answer pairs derived from diverse time series spanning environment, traffic, etc.

The introduced resource asks participants to answer natural-language questions over time series, including explicit numerical analytical tasks, while model pretraining is described as the authors' use of the dataset rather than the benchmark task.

a unified framework that enables natural language queries across multiple time series tasks - numerical analytical tasks and open-ended question answering with reasoning.

we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use

ToolHop evaluates agents on answering user queries via multi-hop tool use, which is neither data analysis nor participant model-building/MLE work allocation.Resource tasks focus on multi-hop tool use/function-calling evaluation rather than analysis workflows or training/tuning predictive models by the evaluated participant.

we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use

Our survey findings and open dataset advance work in accessibility-focused maps, routing algorithms, and urban planning.

The introduced open dataset is for analyzing survey responses about perceived sidewalk barriers and deriving accessibility findings, with no evaluated participant task requiring model building.

This paper presents a large-scale online survey exploring how five mobility groups -- users of canes, walkers, mobility scooters, manual wheelchairs, and motorized wheelchairs -- perceive sidewalk barriers.

ChainBuddy, an AI workflow generation assistant built into the ChainForge platform, aims to tackle this issue.

The introduced resource supports generating evaluative LLM workflows/pipelines, which is an AI workflow construction task rather than data analysis or predictive model-building by evaluated participants.

From a single prompt or chat, ChainBuddy generates a starter evaluative LLM pipeline in ChainForge aligned to the user’s requirements.

We present TeachTune, where teachers can create simulated students and review PCAs by observing automated chats between PCAs and simulated students.

The introduced resource supports teachers reviewing pedagogical conversational agents via simulated student chats, which is neither data analysis nor predictive model-building by the evaluated participant.

We present TeachTune, where teachers can create simulated students and review PCAs by observing automated chats between PCAs and simulated students.

We present BASIS (Bootstrapping Assistant modeling with Situated Interaction Simulation), a novel framework that fundamentally rethinks how visual assistants are developed and evaluated.

BASIS concerns developing and evaluating visual assistants whose delivered task is guiding real users through complex situated tasks, not data analysis or participant model-building benchmark tasks established as the evaluated deliverable here.

synthetic interaction data from simulated users and assistants can boost the development of visual assistant models that effectively guide real users to complete complex tasks.

we develop a comprehensive evaluation framework to examine agents' safety across three progressive dimensions

The introduced evaluation framework asks agents to demonstrate risk knowledge, identify risks, and avoid unsafe actions, which are safety behavior tasks rather than data analysis or predictive model building.

we develop a comprehensive evaluation framework to examine agents' safety across three progressive dimensions: 1) their knowledge about potential risks, 2) their ability to identify corresponding risks in execution trajectories, and 3) their actual behaviors to avoid executing these risky actions.

we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs.

ACEBench evaluates LLM tool usage in basic, ambiguous, and multi-turn dialogue scenarios, which is neither data analysis nor predictive model building.

ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. “Normal” evaluates tool usage in basic scenarios; “Special” evaluates tool usage in situations with ambiguous or incomplete instructions; “Agent” evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues.

we introduce NarrativeFactScore (NFS), the first “Agent-as-a-Judge” framework that evaluates and refines factuality in narrative summarization

The introduced resource is an evaluation/refinement framework for factuality in long narrative summarization, not a data analysis workflow or participant model-building task.

we introduce NarrativeFactScore (NFS), the first “Agent-as-a-Judge” framework that evaluates and refines factuality in narrative summarization

we present **AIRepr**, an **A**nalyst–**I**nspector framework for automatically evaluating and improving the **repr**oducibility of LLM-generated data analysis workflows

AIRepr evaluates reproducibility of LLM-generated data analysis workflows, so the evaluated work is data analysis rather than building predictive models.

we present **AIRepr**, an **A**nalyst–**I**nspector framework for automatically evaluating and improving the **repr**oducibility of LLM-generated data analysis workflows

we extend an existing DST dataset by generating utterances of a second user based on speech act theory.

The extended resource evaluates multi-user dialogue state tracking, which is a dialogue inference task rather than data analysis or participant model-building work, despite involving LLM evaluation and dataset extension methodology by the authors言ى.ע彩争霸:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl:UIControl

we assess the robustness of LLMs in multi-user DST while minimizing dataset construction costs.

We present EnterpriseBench, a comprehensive benchmark that simulates enterprise settings, featuring 500 diverse tasks across software engineering, HR, finance, and administrative domains.

EnterpriseBench evaluates broad enterprise workflow tasks across software engineering, HR, finance, and administration, without metadata showing that participants must perform data analysis or build predictive models.

We present EnterpriseBench, a comprehensive benchmark that simulates enterprise settings, featuring 500 diverse tasks across software engineering, HR, finance, and administrative domains.

We introduce a dataset of 114,296 teacher-learner conversations grounded in 14,287 tutorials across 17 domains and 727 topics.

The introduced resource consists of teacher-learner procedural learning conversations, which is an educational dialogue/procedural learning task rather than data analysis or model-building by evaluated participants.

We introduce a dataset of 114,296 teacher-learner conversations grounded in 14,287 tutorials across 17 domains and 727 topics.

we contribute WIKICOLLIDE, the first benchmark of real Wikipedia inconsistencies.

WIKICOLLIDE evaluates detecting factual inconsistencies in Wikipedia, with no evidence that participants must perform data analysis workflows or build/tune predictive models as the delivered task artifact.Web search result snippets are not valid evidence, but the supplied abstract identifies WIKICOLLIDE as a benchmark of real Wikipedia inconsistencies rather than an analysis or modeling task suite as defined here, and no participant model-building requirement is stated in the metadata. The category is others because the benchmark task is inconsistency detection, an information verification task, not explicit data analysis or MLE model-building delivery by the evaluated participant.

we contribute WIKICOLLIDE, the first benchmark of real Wikipedia inconsistencies.

we present FlowExplainBench, a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types

FlowExplainBench evaluates flowchart attribution, where participants trace components grounding flowchart QA responses, which is neither data analysis nor model-building/MLE work for the evaluated participant.

we present FlowExplainBench, a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types

we clean and restructure the MedCalc-Bench dataset and propose a new step-by-step evaluation pipeline

The resource evaluates medical calculation reasoning steps such as formula selection, entity extraction, and arithmetic computation, not data analysis workflows or model-building by the participant.

step-by-step evaluation pipeline that independently assesses formula selection, entity extraction, and arithmetic computation

we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research

The introduced benchmarks are for GUI agent decision-making in mobile apps, not data analysis or model-building tasks assigned to participants/the evaluated agent (the agent delivers navigation decisions/actions).

Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research.

we introduce PEARL, a large-scale Arabic multimodal dataset and benchmark explicitly designed for cultural understanding

The introduced resources evaluate Arabic multimodal cultural understanding rather than data analysis workflows or participant-built predictive models/code modeling tasks assigned to evaluated agents/users are not established by the metadata.resources appear to be datasets/benchmarks for LVLM cultural understanding, with ordinary model evaluation/instruction examples rather than data analysis or model-building tasks.meta explicitly frames PEARL as a culturally-aware multimodal dataset/benchmark, so the identifiable task is cultural understanding, which falls outside analysis/modeling under the rubric.

we introduce PEARL, a large-scale Arabic multimodal dataset and benchmark explicitly designed for cultural understanding

we develop Perovskite-KG, a domain-specific knowledge graph constructed from 1,517 research papers, containing 23,789 entities and 22,272 relationships

The introduced resources support knowledge retrieval, question answering, and scientific reasoning/problem-solving for perovskite solar cell research, not data analysis workflows or model-building tasks assigned to evaluated participants.

Experimental results demonstrate that our system significantly outperforms existing models in both domain-specific knowledge retrieval and scientific reasoning tasks, providing researchers with effective tools for literature review, experimental design, and complex problem-solving in PSC research.

This study introduces ResearchArena, a benchmark designed to evaluate LLMs’ capabilities in conducting academic surveys

The benchmark evaluates literature discovery, relevance selection, and knowledge organization for academic surveys, which are research-agent information tasks rather than data analysis workflows or model-building tasks.

ResearchArena models the process in three stages: (1) information discovery, identifying relevant literature; (2) information selection, evaluating papers’ relevance and impact; and (3) information organization, structuring knowledge into hierarchical frameworks such as mind-maps.

we construct a novel dataset RETAIL

RETAIL evaluates travel planning and decision-making outputs, not data analysis workflows or participant-built predictive models; the paper's TGMA framework is not the resource task being classified.

we construct a novel dataset RETAIL, which supports decision-making for implicit queries while covering explicit queries, both with and without revision needs.

We distil these capabilities into 3B and 8B models via two-stage LoRA fine-tuning on synthetic GPT-4o-generated data.

The introduced resource supports generating pedagogically valid answers to student questions about feedback reports, which is an educational response-generation task rather than data analysis or participant model-building work; the LoRA fine-tuning is the authors' resource construction method, not the evaluated task to classify as modeling by itself.anjeunnaother_evidence is strongest for the delivered behavior of the introduced SCRIBE framework/dataset use.

designed to generate valid responses to student questions about feedback reports

we introduce a new dataset, CalTab151, specifically designed for complex numerical reasoning over tables.

CalTab151 evaluates participants on complex numerical reasoning over tabular data, where the delivered output is an answer derived from tables rather than a trained predictive model.

we introduce a new dataset, CalTab151, specifically designed for complex numerical reasoning over tables.

We introduce Text2Vis, a benchmark designed to assess text-to-visualization models

Text2Vis evaluates systems that answer data science queries and generate visualizations from tables, including trend, correlation, outlier, and predictive analytics, which are data analysis workflows rather than model-building tasks.

Automated data visualization plays a crucial role in simplifying data interpretation, enhancing decision-making, and improving efficiency.

we present CSUSER, an evaluation protocol with LLM-based user simulator grounded in real-world behavioral data

The introduced CSUSER protocol evaluates conversational sales agents that conduct dialogue for recommendations and persuasion, which is neither data analysis nor model-building by the evaluated participant.

Conversational Recommender Systems (CRSs) aim to engage users in dialogue to provide tailored recommendations.

we introduce APPUI, the first benchmark dataset for AppUI2Code, constructed through a human-in-the-loop process to ensure data quality and coverage.

APPUI evaluates generating/reconstructing front-end code from UI design mockups, which is a software/UI generation task rather than data analysis or model-building by the participant.

the AppUI2Code task, which aims to reconstruct static single-page applications from design mockups

we introduce VisFinEval, the first large-scale Chinese benchmark that spans the full front-middle-back office lifecycle of financial tasks

The introduced benchmark asks evaluated models to answer multimodal financial questions including explicit financial data analysis and decision support, with no evidence that participants must build or train predictive models.

VisFinEval comprises 15,848 annotated question–answer pairs drawn from eight common financial image modalities (e.g., K-line charts, financial statements, official seals), organized into three hierarchical scenario depths: Financial Knowledge & Data Analysis, Financial Analysis & Decision Support, and Financial Risk Control & Asset Optimization.

finetunes a dedicated object detection model for visual analysis. It reduces the requirement for domain-specific training data from millions of samples to a few hundred.

The introduced resource supports an embodied Minecraft agent performing open-world decision-making tasks, while the object-detector finetuning is part of the authors' system construction rather than the evaluated participant's task delivery.

Large language models (LLMs) have shown significant promise in embodied decision-making tasks within virtual open-world environments.

we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games

BALROG evaluates agents acting and reasoning in game environments, not data analysis workflows or participant-built predictive models.l

we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games

we introduce DataEnvGym, a testbed of teacher environments for data generation agents.

The introduced resource evaluates agents that generate training data plans/data in a teacher environment, while student model training is part of the environment feedback loop rather than the participant's delivered task.

DataEnvGym frames data generation as a sequential decision-making task, involving an agent consisting of a data generation policy (which generates a plan for creating training data) and a data generation engine (which transforms the plan into data), inside an environment that provides feedback from a student.

we introduce DSBench, a comprehensive benchmark designed to evaluate data science agents with realistic tasks

DSBench explicitly evaluates agents on both data analysis tasks and end-to-end data modeling tasks.

This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions.

DSBench offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks.

We introduce the MuSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems.

MuSE evaluates multimodal agent systems' task/model selection and action planning rather than data analysis or participant-built predictive modeling workflows अनु implied by the metadata should be classified as other agent capability evaluation tasks한다 .၎င်း Uses a benchmark for planning operations across multiple models, not asking participants to train or tune models themselves . data processing or model use is not enough for analysis/modeling under the rubric . Delete stray multilingual tokens? Need JSON valid. final must be clean. redo final.

We introduce the MuSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems.

We introduce InsightBench, a benchmark dataset with three key features.

InsightBench evaluates agents on end-to-end business data analytics and insight generation, where the participant delivers questions, interpretations, summaries, and actionable insights rather than trained predictive models.

InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps.

LEARN-BY-INTERACT synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction.

The introduced resource supports adapting agents for downstream coding, web, and desktop agentic tasks, not participant-delivered data analysis or model-building tasks as a benchmark resource task itself is not specified as MLE work for evaluated participants/code users beyond agent adaptation data use by the authors/resource consumers).

Extensive experiments on SWE-bench, WebArena, OSWorld, and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of LEARN-BY-INTERACT in various downstream agentic tasks

AmbigSQL, a novel task for disambiguating information-seeking requests for complex SQL generation towards data analysis agents

AmbigSQL asks evaluated conversational agents to clarify information-seeking requests for complex SQL generation in support of data analysis, which is a data querying/analysis workflow rather than model building.

AmbigSQL, a novel task for disambiguating information-seeking requests for complex SQL generation towards data analysis agents

We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering.

MLE-bench evaluates agents on Kaggle-style machine learning engineering competitions where the participant must train models, prepare datasets, and run experiments.

creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments.

create a diverse dataset, ProactiveBench, containing 6,790 events

ProactiveBench evaluates agents on proactively offering assistance in event contexts, which is an interactive assistance/dialogue behavior rather than data analysis or participant model-building work as the benchmark task itself asks for assistance behavior outputs or fine-tuning use of the dataset rather than MLE competition-style deliverables.

ProactiveBench, containing 6,790 events. Finally, we demonstrate that fine-tuning models with the proposed ProactiveBench can significantly elicit the proactiveness of LLM agents.

We introduce Spider 2.0, an evaluation framework comprising $632$ real-world text-to-SQL workflow problems

Spider 2.0 evaluates agents on enterprise text-to-SQL workflows whose deliverables are SQL queries for data transformation and analytics, which are data analysis tasks rather than model-building.

Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics.

It first constructs a large-scale fine-tuning dataset WorkflowBench with 106, 763 samples

WorkflowBench is for workflow orchestration over APIs, where the evaluated output is orchestrated workflows rather than data analysis findings or model-building work by participants.

Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs.

we introduce **CoTA**, a new benchmark to evaluate LLMs on conversational tabular data analysis

CoTA evaluates LLMs on multi-turn conversational tabular data analysis, where the participant delivers data exploration and decision-support analysis rather than trained predictive models.

Conversational Tabular Data Analysis, a collaboration between humans and machines, enables real-time data exploration for informed decision-making.

apply it to a new, challenging Minecraft-based assistance game with over $10^{400}$ possible goals

The introduced Minecraft-based assistance game evaluates assistants helping users complete building tasks, which is an interactive decision-making task rather than data analysis or model-building by the evaluated participant.

In a human study, our AssistanceZero-trained assistant significantly reduces the number of actions participants take to complete building tasks in Minecraft.

We present Gravity-Bench-v1, an environment-based benchmark that challenges AI agents

Gravity-Bench-v1 evaluates agents on scientific discovery tasks requiring experimental data collection and dynamic data analysis rather than building predictive models.

Agents must plan to collect data within an experimental budget and must perform a dynamic form of data analysis and reasoning to solve tasks efficiently.

we create a novel dataset of sequential preferences

The introduced resource supports user-centric multi-turn text-to-image generation and preference interaction research, not data analysis workflows or participant model-building tasks as a benchmark deliverable described in the metadata.

We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems.

we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity.

ResearchBench asks evaluated systems to perform node-masking prediction for research community simulation assessment, which is not a data analysis workflow or participant model-building task itself given the metadata evidence available.

we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity.

We present the Berkeley Function Calling Leaderboard (BFCL), a comprehensive benchmark designed to evaluate function calling capabilities in a wide range of real-world settings.

The introduced benchmark evaluates LLM tool/function calling and agentic behavior, not participant-delivered data analysis or model-building work.

We present the Berkeley Function Calling Leaderboard (BFCL), a comprehensive benchmark designed to evaluate function calling capabilities in a wide range of real-world settings.

The resulting HP-KG encompasses over 40k activity steps across more than 6k household tasks, spanning diverse everyday scenarios.

The introduced HP-KG supports robotic manipulation planning, where the evaluated output is a plan/actions rather than data analysis or building a predictive model.

To address the above problems, we propose Hierarchical Procedural Knowledge Graphs (\textbf{HP-KG}) to enhance LLMs for complex robotic planning while significantly reducing the demand for LLM scale in robotic manipulation.

To evaluate performance, we develop MMArt-Bench, a novel benchmark constructed from real-world user edits.

MMArt-Bench evaluates photo retouching/editing outputs rather than data analysis workflows or participant-built predictive models.

To evaluate performance, we develop MMArt-Bench, a novel benchmark constructed from real-world user edits.

we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test

The introduced datasets evaluate pre-operative GUI action critique/error diagnosis rather than data analysis workflows or participant-built predictive modeling tasks according to the abstract context and prior annotation task summary.

we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic data.

our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities

The introduced benchmark is an AutoML agent benchmark whose tasks are framed around end-to-end machine learning automation across data modalities, so evaluated participants must deliver ML modeling workflows.

when evaluated on our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities

We created a benchmark containing 420 Python repositories with unit tests for evaluation.

The introduced benchmark evaluates building executable test environments and Dockerfiles for code repositories, which is a software engineering environment construction task rather than data analysis or predictive model building.

given a code repository, Repo2Run iteratively builds the Docker image, runs unit tests based on the feedback of the building, and synthesizes the Dockerfile until the entire pipeline is executed successfully

Our code and (pending approval) dataset will be released to support reproducibility and future research in AI-driven education.

The introduced dataset supports automated assessment and feedback for student-drawn scientific sketches, which is an educational evaluation task rather than data analysis or participant model-building work.

Scientific sketches (e.g., models) offer a powerful lens into students' conceptual understanding, yet AI-powered automated assessment of such free-form, visually diverse artifacts remains a critical challenge.

we introduce the Quantitative Reasoning with Data (QRData) benchmark

QRData evaluates participants on answering statistical and causal quantitative reasoning questions grounded in provided real-world data, which is a data analysis task rather than model building.

Quantitative reasoning is a critical skill to analyze data, yet the assessment of such ability remains limited.

we introduce DSEval – a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle

The metadata establishes broad data science agent benchmarking and mentions data analysis/processing, but does not specify whether the introduced tasks require participant-delivered analysis, model-building, or both across the lifecycle.

Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing.

We propose CToolEval, a benchmark designed to evaluate LLMs in the context of Chinese societal applications, featuring 398 APIs across 27 widely-used Apps

CToolEval evaluates LLM agents on tool invocation and task completion in simulated real-world API interactions, not data analysis workflows or model-building tasks.

We further present an evaluation framework that simulates real-life scenarios, to facilitate the assessment of tool invocation ability of LLMs for tool learning and task completion ability for user interation.

The source code and generated dialogue data are available at https://github.com/ParticleMedia/LUAS.

The introduced generated dialogue data is used to support fine-tuning models for DST prediction, which is a predictive modeling task.

Then a two-stage fine-tuning on LLaMA 2 is performed on the generated data and the real data for the DST prediction.

we propose Mobile-Bench, a novel benchmark for evaluating the capabilities of LLM-based mobile agents

Mobile-Bench evaluates LLM-based mobile agents completing mobile app planning and multi-app collaboration tasks, not data analysis workflows or model-building by participants.

Mobile-Bench comprises 832 data entries, with more than 200 tasks specifically designed to evaluate multi-APP collaboration scenarios.

Our benchmark, SportsMetrics, introduces a new mechanism for assessing LLMs’ numerical reasoning and fusion skills.

SportsMetrics tasks require LLMs to perform sports data analytics, including complex data queries and key-statistic analysis, rather than build predictive models.

In this paper, we introduce four novel tasks centered around sports data analytics to evaluate the numerical reasoning and information fusion capabilities of LLMs.

we introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users’ implicit intentions through explicit queries

IN3 evaluates agents on clarifying vague user instructions and understanding implicit intentions, which is an interaction/intention-understanding task rather than data analysis or model building by the participant agent.

a novel benchmark designed to inspect users’ implicit intentions through explicit queries

We evaluate CTIKG on two representative benchmarks built from real world CTI articles

The introduced benchmarks evaluate building security-oriented knowledge graphs from CTI articles, which is information extraction/knowledge graph construction rather than data analysis or model-building by the participant.

We evaluate CTIKG on two representative benchmarks built from real world CTI articles, and the results show that CTIKG achieves 86.88% precision in building security-oriented knowledge graphs, achieving at least 30% improvements over the state-of-the-art techniques.

we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset

The introduced resource benchmarks text-to-diagram generation, where participants deliver generated diagrams rather than data analysis findings or trained predictive models themselves, so it falls under others.

To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset.

We present OpenAgents, an open platform for using and hosting language agents in the wild of everyday life.

The introduced platform includes a data-analysis agent task with Python/SQL and data tools, while the other listed API and web-browsing capabilities do not add predictive model-building tasks.

OpenAgents includes three agents: (1) Data Agent for data analysis with Python/SQL and data tools; (2) Plugins Agent with 200+ daily API tools; (3) Web Agent for autonomous web browsing.

we propose a new executability metric for better measuring the performance of web scraper generation tasks

The introduced evaluation component targets web scraper generation, where the evaluated output is a generated scraper rather than a data analysis workflow or model-building task.

we propose a new executability metric for better measuring the performance of web scraper generation tasks

We introduce DA-Code, a code generation benchmark specifically designed to assess LLMs on agent-based data science tasks.

DA-Code evaluates agents on data science code generation tasks involving real data, data wrangling, analytics, processing, and deriving answers, with no evidence that participants must build predictive models.

examples in DA-Code are all based on real and diverse data, covering a wide range of complex data wrangling and analytics tasks.

Datasets and code are available at: https://github.com/XueyangFeng/ReHAC/.

The resource concerns human-agent collaboration for complex task solving, with no metadata indicating evaluated participants must perform data analysis or build predictive models; the policy training is the paper authors' method, not the resource task itself.

we introduce the problem of LLM-based human-agent collaboration for complex task-solving, exploring their synergistic potential.

a newly created Train-Ticket dataset

The introduced Train-Ticket dataset supports root cause analysis in micro-services architecture, where the evaluated output is identifying root causes and resolutions rather than building predictive models.

Root cause analysis (RCA) in Micro-services architecture (MSA) with escalating complexity encounters complex challenges in maintaining system stability and efficiency due to fault propagation and circular dependencies among nodes.

we have released the first public benchmark dataset for clinical triage with corresponding ESI levels and human expert performance for comparison

The introduced benchmark asks participants to perform clinical triage by assigning ESI levels, which is a diagnosis/classification-style task rather than data analysis or model-building by the evaluated participant, and the paper's TriageAgent framework is not the resource being classified here.

we have released the first public benchmark dataset for clinical triage with corresponding ESI levels and human expert performance for comparison

We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.

AgentBench evaluates LLM agents on interactive reasoning and decision-making environments, but the metadata does not establish that participants must deliver data analysis workflows or build predictive models.

We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.

we introduce CivRealm, an environment inspired by the Civilization game.

CivRealm evaluates agents on interactive game decision-making and reasoning rather than delivering data analysis or building predictive models.

Overall, CivRealm stands as a unique learning and reasoning challenge for decision-making agents.

We propose a formal evaluation framework for assessing the compatibility of near-optimal AI with interaction partners who may have much lower levels of skill

The introduced resource evaluates collaborative chess decision-making with lower-skill partners, not data analysis or model-building tasks assigned to participants.

two chess game frameworks designed to foster collaboration between powerful AI agents and less-skilled partners

We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments.

The introduced Habitat 3.0 resource evaluates embodied collaborative robot tasks such as navigation/following and scene rearrangement, not data analysis or model-building by the evaluated participant.

Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene.

we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools

The introduced benchmark asks evaluated LLMs to decide whether to use tools and select appropriate tools, which is neither data analysis nor model-building work by the participant.

we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools

We introduce MINT, a benchmark that evaluates LLMs' ability to solve tasks with multi-turn interactions

MINT evaluates LLMs on multi-turn reasoning, coding, and decision-making with tools and feedback, not on delivering data analysis workflows or building predictive models/adapting ML systems as the participant task after the resource is introduced or extended by the paper authors themselves.лды өгөгдсөн юм бол энэ нь others ангилал юм, учир нь benchmark нь LLM-ийн multi-turn interaction чадварыг reasoning, coding, decision-making даалгавруудаар үнэлдэг бөгөөд оролцогчоос data analysis insight эсвэл predictive model training/tuning deliver хийхийг шаардаагүй байна. Тиймээс зөв ангилал: others. Нотлох ишлэл нь resource task scope-ийг шууд зааж байна: “We repurpose a diverse set of established evaluation datasets focusing on reasoning, coding, and decision-making and carefully curate them into a compact subset for efficient evaluation.” ёстой байсан юм. Өмнөх annotation нь MINT benchmark/evaluation framework гэж баталж байгаа ч ангиллын нотолгоо metadata-аас авсан байх шаардлагатай бөгөөд энэ ишлэл хангалттай байна. Confidence 5, учир нь abstract resource болон task types-ийг тодорхой өгсөн байна.

We repurpose a diverse set of established evaluation datasets focusing on reasoning, coding, and decision-making and carefully curate them into a compact subset for efficient evaluation.

We introduce SocioDojo, an open-ended lifelong learning environment

SocioDojo evaluates agents on societal analysis and decision-making using real-world text and time series, without requiring participants to deliver trained predictive models.

We introduce SocioDojo, an open-ended lifelong learning environment for developing ready-to-deploy autonomous agents capable of performing human-like analysis and decision-making on societal topics such as economics, finance, politics, and culture.

Our data and code are open-sourced at https://github.com/guosyjlu/DS-Agent.

The introduced DS-Agent resource targets automated data science tasks where the evaluated agent must build and train machine learning models.

In this work, we investigate the potential of large language models (LLMs) based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models.

we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks

The introduced benchmark evaluates agents on end-to-end data analysis questions derived from CSV files, with no participant task of building predictive models.

we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks

introducing realistic features to WebShop, including user profiles demonstrating intentions, personalized reranking reflecting complex environmental dynamics, and runtime cost statistics as self-constraints

The extended WebShop resource evaluates autonomous agents performing reasoning, decision-making, and environmental interaction rather than data analysis or model-building tasks assigned to participants/research agents themselves.

autonomous agents, which leverage the universal capabilities of foundation models to conduct reasoning, decision-making, and environmental interaction

We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements

The introduced benchmark evaluates LLM agents' economic rationality/decision behavior and report-card scores, not data analysis workflows or participant-built predictive models.

We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "rationality report card".

We first build a scalable and parallelizable Android learning environment equipped with a VLM-based general-purpose evaluator

The introduced or extended resource concerns agents performing in-the-wild GUI device control actions, not delivering data analysis findings or building predictive models as the evaluated task itself.

decision-making tasks such as in-the-wild device control through Graphical User Interfaces (GUIs)

construct a new dataset, SocraTeach, which contains $35$K meticulously crafted Socratic-style multi-round (equivalent to $208$K single-round) teaching dialogues

The introduced resource concerns Socratic-style math teaching dialogues and evaluation of LLM teaching quality, which is an educational dialogue/teaching task rather than data analysis or predictive model building by the evaluated participant.

SocraticLM, which achieves a Socratic "Thought-Provoking" teaching paradigm that fulfills the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery.

we first construct the knowledgeable instruction set for UrbanKGC tasks

The introduced instruction set targets urban knowledge graph construction tasks like relational triplet extraction and knowledge graph completion, which are KG construction/inference tasks rather than data analysis workflows or participant model-building tasks.

we first construct the knowledgeable instruction set for UrbanKGC tasks (such as relational triplet extraction and knowledge graph completion) via heterogeneity-aware and geospatial-infused instruction generation.

we provide an extractive summarization dataset covering 1K long-form answers and our user study annotations

The introduced resource supports extractive summarization of long-form answers, where the participant delivers concise summaries rather than data analysis findings or trained predictive models.

we provide an extractive summarization dataset covering 1K long-form answers and our user study annotations

we introduce a new dataset, FactKG: Fact Verificationvia Reasoning on Knowledge Graphs.

FactKG asks evaluated systems to perform KG-based fact verification/reasoning over natural-language claims, not data analysis workflows or model-building tasks.

It consists of 108k natural language claims with five types of reasoning: One-hop, Conjunction, Existence, Multi-hop, and Negation.

we present FolkScope, an intention knowledge graph construction framework

The introduced resource concerns constructing an e-commerce intention knowledge graph, which is knowledge graph construction rather than data analysis or predictive model-building by evaluated participants.

we present FolkScope, an intention knowledge graph construction framework, to reveal the structure of humans’ minds about purchasing items.

Four emotional support metrics are proposed to evaluate the mixed-initiative interactions.

The introduced resource is an evaluation framework for mixed-initiative emotional support dialogue interactions, where the evaluated output is conversational behavior rather than data analysis or model-building.

Four emotional support metrics are proposed to evaluate the mixed-initiative interactions.

we also extract targetoriented dialogues from the chit-chat corpus with a knowledge graph

The introduced resource supports target-oriented dialogue generation, where the evaluated participant delivers dialogue behavior rather than data analysis or model-building work as the task output comes from the resource metadata only as dialogue guidance/reaching a target dialogue task rather than MLE assigned to participants (training data construction is the authors' workflow).ብThe introduced/extended dataset concerns target-oriented dialogue, not data analysis or participant-delivered predictive model building.ብThe introduced/extended resource is for target-oriented dialogue, where the task is guiding conversation to a target rather than data analysis or model-building by evaluated participants.ብThe introduced resource is target-oriented dialogue data for training dialogue systems, where the evaluated task is dialogue guidance/reaching a target rather than data analysis or participant model-building.ብThe resource concerns target-oriented dialogue generation/reaching targets in conversation, while the dataset extraction and model training described are the authors' workflow rather than an evaluated participant task.ብThe resource supports target-oriented dialogue, where the delivered output is a dialogue that reaches a target rather than analytical findings or a trained predictive model.ብThe introduced dataset supports target-oriented dialogue generation, which is neither a data analysis workflow nor a participant-delivered model-building task.ብThe introduced resource supports target-oriented dialogue, where the expected output is a guided conversation to a target rather than analytical insights or a trained predictive model.ብThe introduced dataset supports target-oriented dialogue, an ordinary dialogue-generation task rather than data analysis or participant model-building.

Target-oriented dialogue guides the dialogue to a target quickly and smoothly.

the study preregistration, and all supplemental materials required to reproduce the study are available on OSF

The released study materials support reproducing a collaborative setting framed as data analysis/visualization rather than model building.

Many collaborative data analysis situations benefit from collaborators utilizing different platforms.

we collect a dataset of over 1000 examples

The introduced dataset supports an NLP intent-translation task where systems output goals and constraints from language, not data analysis findings or participant-built predictive models.

In this paper, we build a computational interface capable of translating unstructured language strategies into actionable intent in the form of goals and constraints.

we create a new dataset called Integrative Negotiation Dataset (IND).

The introduced dataset supports integrative negotiation dialogue agents whose task is negotiating deal terms, not data analysis or participant-built predictive modeling.

Our dialogue agent is integrative in nature i.e, it possesses the capability to negotiate on price as well as other factors, such as the addition or removal of items from a deal bundle, thereby offering a more flexible and comprehensive negotiation experience.

we release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent

The introduced resource supports task-oriented dialogue/query rewriting, where the evaluated output is a rewritten dialogue query rather than data analysis or model-building.

Supported by this data, we propose the novel task of multi-user contextual query rewriting: to rewrite a task-oriented chat between two users as a concise task-oriented query that retains only task-relevant information and that is directly consumable by the dialogue system.

We introduce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements in large language models (LLMs).

SalesOps evaluates conversational recommender agents whose deliverable is dialogue with recommendations and educational value, not data analysis or model-building by the participant/agent itself.

provide both product recommendations and educational value through mixed-type mixed-initiative dialog

we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language

The introduced benchmark asks evaluated systems to solve natural-language graph reasoning problems, not to perform data analysis workflows or build predictive models for evaluation.

NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks.

we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs

CogEval evaluates LLM cognitive maps and planning abilities, which are reasoning/planning evaluation tasks rather than data analysis or participant model-building tasks.

we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs