Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation

A repository containing the original code and annotated data for the LREC 2026 paper "Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation " by Lu Xu, Luca Moroni, Roberto Navigli.

🛠️ Installation

Installation from source:

git clone https://github.com/SapienzaNLP/cultural-ea-mt
cd cultural-ea-mt
conda create -n cultural-ea-mt python==3.12
conda activate cultural-ea-mt
pip install -r requirements.txt

Annotated Data

The Annotated validation set of XCTranslate benchmark for Italian and Chinese is reported under: data/references/validation/annotation.

The XCTranslate dataset was released as a SemEval-2025 shared task.

We annotated each sample with one of three labels:

• Culturally Agnostic (0): Entities whose recognition does not require cultural knowledge.
• Culturally Sensitive (1): Entities with clear cultural origin but widely recognized internationally.
• Culturally Local (2): Entities requiring deep local or insider cultural knowledge, meaningful primarily within their own cultural context.

An example from the Italian set:

wikidata_id	entity_types	source_titles	en_titles_0	annotator_0	annotator_1	annotator_2	majority_vote
Q10357223	Person;Fictional entity	Regina Bianca	White Queen	1	1	2	1

Reproducibility

Under the src folder, we provide the python code to:

1. Augment the translation items:
   • Annotate the items with gold Wikidata labels: src/gold_process.py
   • Annotate the items with relik linked Wikidata labels: src/relik_process.py
2. Generate the outputs for a specific LLM.
   • Run models locally: src/translate_local.py
   • Run models through API: src/translation_online.py
3. Run the evaluation of the outputs generated with an LLM using:
   • COMET: semantic score for the generated translation against a reference.
     • code: src/comet_eval.py
   • ETA: exact match for a given translated entity, against gold references.
     • code: src/entity_eval.py

Augment the inputs:

Augment the validation set of XCTransalte with gold retrieved Wikidata labels

python src/gold_process.py

Generate the outputs:

Generate the outputs for a given model model_name and choosen inference_type.

Local

Parameters:

• model_name: huggingface id.
• inference_type: can be: baseline, relik, or gold.

python src/translate_local.py --model_name sapienzanlp/Minerva-7B-instruct-v1.0 --inference_type baseline

Online

Parameters:

• model_name: can be: gpt, gemini, or qwen.
• inference_type: can be: baseline, relik, or gold.

python src/translation_online.py --model_name gpt --inference_type baseline

Evaluate the models:

COMET

Evaluate model with COMET

Parameters:

• model_name: can be: huggingface id (for local model) or the id of the online models.
• output_folder: path to the output folder.

python src/comet_eval.py --model_name sapienzanlp/Minerva-7B-instruct-v1.0 --output_folder /path/to/output/folder

ETA

Evaluate model with ETA

Parameters:

• model_name: can be: huggingface id (for local model) or the id of the online models.
• output_folder: path to the output folder.
• category_splitted: boolean value, if True the metrics will be splitted by wikidata categorie types.

python src/entity_eval.py --model_name sapienzanlp/Minerva-7B-instruct-v1.0 --output_folder /path/to/output/folder

Cite this work

If you use any part of this work, please consider citing the paper as follows:

@inproceedings{xu-etal-2026-cultural,
  title = {Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation},
  author = {Xu, Lu and Moroni, Luca and Navigli, Roberto},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {8794--8812},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  doi = {10.63317/3jxgnspt4srr},
  abstract = {Large Language Models (LLMs) demonstrate strong multilingual capabilities yet exhibit systematic cultural biases that affect entity-aware machine translation. While external knowledge integration improves translation accuracy, the extent of these benefits across varying degrees of cultural specificity remains unexplored. We propose a three-level cultural specificity framework: Culturally Agnostic, Culturally Sensitive, and Culturally Local, to systematically analyze how cultural context affects entity translation difficulty and the utility of external knowledge. Through experiments spanning 11 LLMs and 10 languages, we demonstrate that external knowledge provides substantially greater improvements for culturally local entities (up to 70% in m-ETA) compared to culturally agnostic ones. Our analysis reveals distinct behavioral patterns across model tiers: closed and open-weight models show synergistic improvements in both entity accuracy and overall translation quality, while open-data models struggle with instruction-following despite improved entity accuracy.}
}

🪪 License

The data and software are licensed under Creative Commons Attribution-ShareAlike 4.0 International License.

Acknowledgements

The authors gratefully acknowledge the support of the AI Factory IT4LIA project.