Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains

arXiv preprint, 2026

Read the paper DOI 10.48550/arXiv.2603.14400

Summary

This paper extends surprisal-based minimal-pairs evaluation of language models from binary grammaticality judgments to ordinal-scaled classification and scoring tasks. Instead of asking models to generate answers, it measures the surprisal a model assigns to each position on a rating scale, yielding surprisal curves that reveal both the model's preferred response and its uncertainty via entropy, demonstrated across four applied domains including social-ecological-technological systems classification and deductive qualitative coding.

Cite this paper

Katz, A. (2026). Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains. arXiv preprint. https://doi.org/10.48550/arXiv.2603.14400

BibTeX

@misc{katz2026extending,
  title = {Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains},
  author = {Katz, Andrew},
  journal = {arXiv preprint},
  year = {2026},
  doi = {10.48550/arXiv.2603.14400}
}

All publications