Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization
Shaohua Duan, Pengcheng Huang, Xinze Li, Zhenghao Liu, Xiaoyuan Yi, Yukun Yan, Shuo Wang, Yu Gu, Ge Yu, Maosong Sun
ACL 2026
Proceedings · arXiv v2 · Code · Hugging Face · ORKG · Full text (arXiv HTML)
LongMab uses multi-armed bandit sampling to construct preference pairs for long-context language models. A context is divided into chunks, and each chunk is treated as a bandit arm.
How the sampling works
During rollouts, an upper-confidence-bound policy balances exploring under-sampled chunks and selecting chunks with higher estimated rewards. Feedback from the generated responses updates the chunk estimates. The sampled responses are then used to build pairs for direct preference optimization. Source
What the experiments evaluate
The paper evaluates sample construction and downstream performance on long-context tasks with Llama and Qwen models. The evaluation and ablations describe the effect of bandit-guided sampling on the preference data and the trained models. Source
Constructing preference pairs for long-context DPO
| Approach | How it constructs preferences |
|---|---|
| LongReward | Scores candidate responses with an LLM that assesses helpfulness, logicality, faithfulness, and completeness. |
| LOGO | Constructs long-context preference data using position synthesis and trains with reference-free preference optimization. |
| LongMab-PO | Treats document chunks as bandit arms and adapts which chunks to sample during response generation, then uses those responses to construct DPO pairs. |
Section 4 describes the comparison settings. Section 5 separates downstream long-context QA results from ablations on candidate sampling and preference data construction.
Published citation
The ACL 2026 proceedings and arXiv v2 list the current authors. Use the proceedings entry for the publication year, page range, and DOI.