Choosing papers for LLM experiments

Pengcheng Huang · 中文版

When a system gives a bad answer, the experiment you need depends on where it failed. These notes pair common symptoms with a comparison you can run and papers that support different claims. I coauthored ExpandR, ParamMute, SPARK, LongMab-PO, ToolMaster, EC²-Bench, Translate-and-Revise and UNCODE.

Suppose a user types “refund deadline” and the relevant passage says “returns are accepted within thirty days.” Inspect the retrieved passages before judging the generator. Compare the original query with an expanded one against the same corpus, relevance labels and top-k cutoff. An expansion that adds the wrong product name can make a short query worse.

Query2doc is a baseline for adding a generated pseudo-document with a fixed retriever. ExpandR addresses a different experimental choice: when the dense retriever can also be trained, use ranking feedback to teach the generator which expansions help. The paper’s ablations separate the effects of training the retriever and the expansion model. Query2doc · ExpandR, §5.2

The evidence is in the prompt, but the RAG answer is wrong

Keep three items for each error: the answer without retrieval, the passage actually passed to the model, and the answer with retrieval. A museum changing its closing time gives two different tests. Does the model abandon the old time when shown an accurate update? Would it also abandon the correct time for a passage containing a typo? Combining those outcomes into one “follows context” score rewards some mistakes.

CoFaithfulQA, introduced in ParamMute, identifies conflicts where an original, accurate passage disagrees with a model’s stable closed-book answer. ClashEval deliberately alters information in a passage. ParamMute studies suppressing knowledge-critical feed-forward networks; SPARK learns to vary their contribution during generation and separately examines reliable and misleading evidence. Cite the benchmark that matches the direction of your conflict. ParamMute and CoFaithfulQA · ClashEval · SPARK, §6.3

A model accepts a long document but overlooks the decisive paragraph

Move an answer-bearing passage between the beginning, middle and end while keeping the question fixed. If accuracy changes with position, report that dependence. A separate training question is how to obtain two plausible answers of different quality when every rollout can see a long input.

LongReward supplies AI feedback on candidate answers. LongMab-PO changes how candidates are obtained: it samples document chunks, updates their sampling scores from rollout rewards, and uses the resulting answers to construct preference pairs. Compare these choices at the same candidate budget if the claim concerns preference-data quality. LongReward · LongMab-PO, method and ablations

An agent can call familiar tools but fails on a new API

A well-formed call can still use the wrong argument or do the wrong thing. To test transfer, hold out tools, examine their similarity to the training tools, and score whether the task actually succeeds. A useful comparison fixes the number of calls and varies whether the agent can spend one probing an unfamiliar tool.

ToolLLM trains on instruction and API-solution data. ToolMaster studies learning to trial tools and correct errors from execution feedback before completing the task. ToolHop provides multi-hop tasks for testing tool use; ToolMaster reports familiar and unfamiliar subsets in its evaluation. ToolLLM · ToolMaster, Table 2 · ToolHop

A local assistant is cheap to run, but occasionally needs a cloud model

A request router chooses a model for an incoming prompt. An agent filling out a spreadsheet has a second decision: it could ask the cloud for help at one step and finish the rest locally. Compare full trajectories at matched task difficulty. Record completion, cloud cost, and which annotated sensitive items became visible to the cloud; the number of calls alone misses what each call disclosed.

RouteLLM is a reference for the prompt-level quality–cost trade-off. EC²-Bench compares routing, delegation and advisory strategies on 128 executable tasks, including 100 with privacy annotations. Its exposure metric describes information in the cloud model’s context; device latency requires its own measurement. RouteLLM · EC²-Bench, evaluation

A translation reads well but misses a required glossary term

First check whether the required source–target pair reached the model. Then inspect the draft for an acceptable inflected form. If the source says “Save the draft” and the glossary requires brouillon, “Enregistrez le document” is fluent but loses the distinction. A reviewer can flag just that pair, request a revision of the sentence, and check it in context.

For a comparison of constrained decoding and terminology prompting, see the WMT terminology study. Translate-and-Revise evaluates giving the model feedback on unmet constraints after its first translation. Report term compliance alongside translation quality; each can change while the other stays similar. WMT terminology study · Translate-and-Revise, experiments

Two diffusion decoders give different reasoning results on the same model

Hold the checkpoint, generation length and forward-pass budget fixed. Record which positions each decoder unmasks first as well as whether the final answer is correct. This separates a difference in token order from a difference in compute.

LLaDA provides a masked diffusion language model and sampling baseline. UNCODE studies how confidence-based selection favors certain positions and easy tokens, then tests training-free corrections to those choices. Its analysis of token-selection trajectories is relevant when the question is why two decoding policies diverge. LLaDA · UNCODE, decoding-bias analysis