CoFaithfulQA
CoFaithfulQA contains 8,529 question–passage examples from six QA sources. It supports evaluation of retrieval-augmented generation when an accurate passage conflicts with the answer a language model gives from memory.
What does one row measure?
The passage in context supports an answer in answers. The parametric_answer field records the answer the model used to construct the benchmark gave without that passage. Its frequency across repeated generations is recorded in parametric_answer_freq.
For a new model, first answer the same question without the passage. A disagreement identified for the original model may disappear for yours. Report how many disagreements remain before measuring how often the retrieved passage resolves them. The Hub groups the six files under a split called train. The paper uses the examples for evaluation. The download counts below describe the raw public files. Table 1 in the paper reports counts after deduplication.
The source-specific Croissant descriptor lets a loader select individual record sets, such as hotpotqa or newsqa, and verify the checksum of the original file.
Benchmark construction and evaluation in the paper
Data files
Hugging Face dataset card · The downloads below use a fixed revision.
- HotpotQA · 1,476 rows
- NewsQA · 934 rows
- Natural Questions · 1,479 rows
- SearchQA · 1,497 rows
- SQuAD · 2,351 rows
- TriviaQA · 792 rows
Related paper
ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation · Paper on OpenReview