Formalized Information Needs Improve Large-Language-Model Relevance Judgments Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, and Philipp Schaer. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26).
Tip
The camera ready version is now available on arXiv.
Cranfield-style retrieval evaluations with too few or too many relevant documents or with low inter-assessor agreement on relevance can reduce the reliability of observations. In evaluations with human assessors, information needs are often formalized as retrieval topics to avoid an excessive number of relevant documents while maintaining good agreement. However, emerging evaluation setups that use Large Language Models (LLMs) as relevance assessors often use only queries, potentially decreasing the reliability. To study whether LLM relevance assessors benefit from formalized information needs, we synthetically formalize information needs with LLMs into topics that follow the established structure from previous human relevance assessments (i.e., descriptions and narratives). We compare assessors using synthetically formalized topics against the LLM-default query-only assessor on the 2019/2020 editions of TREC Deep Learning and Robust04. We find that assessors without formalization judge many more documents relevant and have a lower agreement, leading to reduced reliability in retrieval evaluations. Furthermore, we show that the formalized topics improve agreement between human and LLM relevance judgments, even when the topics are not highly similar to their human counterparts. Our findings indicate that LLM relevance assessors should use formalized information needs, as is standard for human assessment, and synthetically formalize topics when no human formalization exists to improve evaluation reliability.
Install the dependencies listed in the pyprojects.toml:
uv sync
All synthesized topics and LLM judgments are provided in the data/interim directory. Summarizing statistics such as topic and label similarity are provided in the data/processed directory.
The Topic-Gen toolkit is used to generate topics and qrels.
Topics are generated with the gen-topics.py script.
Topic generation:
- query: Query(-variants) only
- docs-pos: Relevant Documents
- docs-neg: Non-Relevant Documents
- query-docs-pos: Query(-variants) + Relevant Documents
- query-docs-neg: Query(-variants) + Non-Relevant Documents
- contrastive: Relevant + Non-Relevant Documents
- query-contrastive: Query(-variants) + Relevant + Non-Relevant Documents
Masked topic generation:
- topic-masked-title
- topic-masked-description
- topic-masked-narrative
- topic-masked-title-description
- topic-masked-title-narrative
- topic-masked-description-narrative
Qrels are generated with the gen-qrels.py script.
Qrel generation:
Masked qrel generation:
- -DNA-zero-shot-masked-title
- -DNA-zero-shot-masked-description
- -DNA-zero-shot-masked-narrative
- -DNA-zero-shot-masked-title-description
- -DNA-zero-shot-masked-title-narrative
- -DNA-zero-shot-masked-description-narrative
All evaluation scripts are available from the scripts directory.