Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction

Xingjie Gao, Pengcheng Huang, Zhenghao Liu, Yukun Yan, Shuo Wang, Zulong Chen, Chen Qian, Ge Yu, Yu Gu

arXiv preprint

arXiv · Code · Hugging Face · ORKG · Full text (arXiv HTML)

ToolMaster studies how language-model agents learn to use unfamiliar tools by interacting with an environment. A trial phase lets the agent probe available tools and use feedback before executing a task.

Training for tool trialing and execution

The supervised stage uses teacher trajectories that include explicit tool trials and self-corrections. A subsequent stage uses Group Relative Policy Optimization to train trialing and execution jointly. The paper evaluates whether the learned behavior transfers to previously unseen tools. Source

Learning to call previously unseen tools

Approach Where the agent learns about an unfamiliar tool
ToolLLM Collects successful multi-step API solutions with a search procedure, then uses the resulting instruction data to train a tool-using model.
Simulated Trial and Error (STE) Simulates useful situations for a tool, interacts with it, and stores lessons from the execution feedback for later use.
ToolMaster Trains an explicit trial phase before task execution. Supervised trajectories model tool probes and self-correction, followed by reinforcement learning over trial and execution together.

The table compares how methods acquire tool experience. Each paper uses its own evaluation setting.

Evaluating generalization on ToolHop

ToolHop pairs multi-hop questions with interdependent, locally executable tools. ToolMaster uses it as an out-of-domain test. Its Table 2 reports answer correctness on ToolHop and Solvable Pass Rate on TMDB. The ToolHop dataset card documents the original benchmark and its evaluation settings.

Section 5.4 and Figure 4 compare ToolMaster with an ablation trained on successful tool-use trajectories. The analysis groups ToolHop tools by how similar their documentation is to that of tools seen during training. A query is treated as unfamiliar if any required tool falls into the low-similarity group, and as familiar if every required tool falls into the high-similarity group.

Released trajectories and checkpoint

The released SFT file contains 1,444 query records, each with one conversation. Section 4 reports selecting 1,500 ToolBench queries to construct the SFT dataset. The two counts refer to the released file and the paper’s query selection, respectively. The ToolMaster-7B checkpoint is linked from the official repository.

DOI 10.48550/arXiv.2601.12762 · BibTeX