Understanding inverse document frequency: on theoretical arguments for IDF

Stephen Robertson (Microsoft Research, Cambridge, UK and City University, London, UK)

Journal of Documentation

ISSN: 0022-0418

Publication date: 1 October 2004

Abstract

The term‐weighting function known as IDF was proposed in 1972, and has since been extremely widely used, usually as part of a TF*IDF function. It is often described as a heuristic, and many papers have been written (some based on Shannon's Information Theory) seeking to establish some theoretical basis for it. Some of these attempts are reviewed, and it is shown that the Information Theory approaches are problematic, but that there are good theoretical justifications of both IDF and TF*IDF in the traditional probabilistic model of information retrieval.

Keywords

Citation

Robertson, S. (2004), "Understanding inverse document frequency: on theoretical arguments for IDF", Journal of Documentation, Vol. 60 No. 5, pp. 503-520. https://doi.org/10.1108/00220410410560582

Download as .RIS

Publisher

:

Emerald Group Publishing Limited

Copyright © 2004, Emerald Group Publishing Limited

To read the full version of this content please select one of the options below

You may be able to access this content by logging in via Shibboleth, Open Athens or with your Emerald account.
To rent this content from Deepdyve, please click the button.
If you think you should have access to this content, click the button to contact our support team.