Overview of the INEX 2008 Ad Hoc Track
Jaap Kamps, Shlomo Geva, Andrew Trotman, Alan Woodley, and Marijn
Koolen
This paper gives an overview of the INEX 2008 Ad Hoc Track.
The main goals of the Ad Hoc Track were two-fold. The first goal was to
investigate the value of the internal document structure (as provided by the
XML mark-up) for retrieving relevant information. This is a continuation of
INEX 2007 and, for this reason, the retrieval results are liberalized to
arbitrary passages and measures were chosen to fairly compare systems
retrieving elements, ranges of elements, and arbitrary passages. The second
goal was to compare focused retrieval to article retrieval more directly than
in earlier years. For this reason, standard document retrieval rankings have
been derived from all runs, and evaluated with standard measures. In addition,
a set of queries targeting Wikipedia have been derived from a proxy log, and
the runs are also evaluated against the clicked Wikipedia pages. The INEX 2008
Ad Hoc Track featured three tasks: For the Focused Task a ranked-list of nonoverlapping results (elements or passages) was needed.
For the Relevant in Context Task non-overlapping results (elements or passages)
were returned grouped by the article from which they came. For the Best in Context
Task a single starting point (element start tag or passage start) for each
article was needed. We discuss the results for the three tasks, and examine the
relative effectiveness of element and passage retrieval. This is examined in
the context of content only (CO, or Keyword) search as well as content and
structure (CAS, or structured) search. Finally, we look at the ability of
focused retrieval techniques to rank articles, using standard document
retrieval techniques, both against the judged topics as well as against queries
and clicks from a proxy log.