Findings of ACL 2025 Conference paper

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

Studying how the spacing between related facts changes a language model’s ability to use a long context.

Runchu Tian^*, Yanghao Li^*, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu

* Equal contribution.

LongPiBench construction, positional data augmentation, and task examples. — Fig. 2 — Original LongPiBench construction and task examples.Original source ↗

01 / The question

Does the distance between relevant facts matter, beyond their individual positions in a long input?

02 / The approach

Long-context positional bias

LongPiBench varies the placement and spacing of multiple relevant information pieces across tasks and input lengths. The study distinguishes absolute position from relative distance to reveal biases that single-fact retrieval tests can miss.

Read the full paper

03 / Citation

Cite this work

BibTeX

@inproceedings{tian-etal-2025-distance,
    title = "Distance between Relevant Information Pieces Causes Bias in Long-Context {LLM}s",
    author = "Tian, Runchu  and
      Li, Yanghao  and
      Fu, Yuepeng  and
      Deng, Siyang  and
      Luo, Qinyu  and
      Qian, Cheng  and
      Wang, Shuo  and
      Cong, Xin  and
      Zhang, Zhong  and
      Wu, Yesai  and
      Lin, Yankai  and
      Wang, Huadong  and
      Liu, Xiaojiang",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.28/",
    doi = "10.18653/v1/2025.findings-acl.28",
    pages = "521--533",
    ISBN = "979-8-89176-256-5"
}

Select the citation to copy it manually.

04 / Versions & records

Publication record

1 Google Scholar record

Distance between relevant information pieces causes bias in long-context LLMs ↗

READ HERE · 13 PAGES

Full paper

Document

Loading document…

Scroll through every page. Select text to copy, or use search to find a passage.

More research

View all

Does the distance between relevant facts matter, beyond their individual positions in a long input?

Long-context positional bias

Cite this work

Publication record

Full paper

More research

One-Shot On-Policy Distillation↗

Boundary-Certificate Proofs↗

One-Shot On-Policy Distillation

Boundary-Certificate Proofs