SIGIR 2004 Workshop: RIA and "Where Can IR Go From Here?"
In the summer of 2003, NIST organized a 6-week workshop called Reliable Information Access or RIA (sponsored by ARDA). The RIA workshop brought together seven different top research IR systems and investigated how the systems were getting their results, and why the systems failed on some topics and succeeded on others. The SIGIR 2004 workshop will focus on discussing the implications of lessons learned from RIA; how they affect our understanding of what is currently happening in research systems, and what they suggest are areas of IR research that warrant immediate concentrated work. Some of the most important lessons have to do with how to study IR systems, and how to make use of multiple IR systems to sharpen our understanding of the principles behind the performance observed. This is a significant change in IR research over the "single lab model" which has been dominant up to now. One of the major products of the RIA workshop is a massive (over 40 GBytes) database, including intensive manual failure analyses on 45 TREC topics and why each system failed on those topics. The database also includes over 3,000 total runs done by these research systems over 9 different TREC test collections. This database will be publicly available after the SIGIR workshop as a resource for the research community. Because the main goal of the workshop is presentation to a wider audience of what occurred in RIA, and lots of discussion with the audience, the talks are all given by RIA workshop participants. Each of these participants will be presenting their work at the workshop, their work since that workshop, and a list of questions to the audience to provoke discussion of how others see the issues.

Tentative Agenda

Thursday, July 29 9-9:30 a.m. Introduction to the 2003 RIA workshop Donna Harman (NIST) 9:30-10:30 a.m. Review of failure analysis work Chris Buckley (SabIR) 10:30-11 a.m. Break 11-11:45 a.m. Review of the feedback experiments involving the number of documents used (bf_numdocs, bf_numdocs_relonly experiments) Jesse Montgomery/David Evans (CLARIT) 11:45 a.m.-12:30 p.m. Review of the feedback experiments involving the number and source of terms used (bf_numterms, bf_pass_numterms experiments) Paul Ogilvie (Carnegie Mellon U.) 12:30-1:30 p.m. Lunch 1:30-2:15 p.m. Review of the feedback experiments involving the swapping of documents or terms across systems (bf_swap_doc/term,doc_cluster,doc_fuse) Tom Lynam/Gord Cormack (U. of Waterloo) 2:15-3 p.m. Tour of new web site, including open discussion of ways to use it and improve it Chris Buckley (SabIR) 3 p.m. on discussion groups (probably one group) possible topics: what RIA tells us about likely ways of tackling expansion experiments, what people discovered about their systems. possible topics: what RIA tells us about likely ways of tackling expansion experiments, what people discovered about their systems. Return to Projects and Activities Homepage Return to Retrieval Group Homepage
Last updated: Friday, 02-Jul-2004 18:15:42 UTC Date created: Friday, July 2, 2004 trec@nist.gov