Abstract
Purpose
The purpose of this paper is to examine changes in relevance assessments, specifically the selection of relevance criteria by subjects as they move through the information search process.
Design/methodology/approach
The paper examines the relevance criteria choices of 39 subjects in relation to search stage. Subjects were assigned a specific search task in a controlled test. Statistics were collected and analyzed using descriptive statistics and the chi‐square goodness‐of‐fit tests.
Findings
The statistically significant findings identified a number of commonly reported relevance criteria, which varied over an information search process for relevant and partially relevant judgments. These results provide statistical confirmations of previous studies, and extend these findings identifying specific criteria for both relevant and partially relevant judgments.
Research limitations/implications
The study only examines a short duration search process and since the convenience sample of subjects were from similar backgrounds and were assigned similar tasks, the study did not explicitly examine the impact of contextual factors such as user experience, background or task in relation to relevance criteria choices.
Practical implications
The paper has implications for the development of search systems which are adaptive and recognize the cognitive changes which occur during the information search process. Examining and identifying relevance criteria beyond topicality and the importance of those criteria to a user can help in the generation of better search queries.
Originality/value
The paper adds more rigorous statistical analysis to the study of relevance criteria and the information search process.
Keywords
Citation
Taylor, A., Zhang, X. and Amadio, W.J. (2009), "Examination of relevance criteria choices and the information search process", Journal of Documentation, Vol. 65 No. 5, pp. 719-744. https://doi.org/10.1108/00220410910983083
Publisher
:Emerald Group Publishing Limited
Copyright © 2009, Emerald Group Publishing Limited
Introduction
Relevance is a foundational concept for the study of information retrieval (IR) systems. The primary metrics used in IR, recall and precision, are based on a dichotomous concept of relevance where a document is either relevant or not. But studies (Spink et al., 1998; Tang and Solomon, 1998) have indicated that dichotomous relevance does not adequately describe or measure the cognitive processes involved in selecting documents as part of an IR search.
Research (Barry, 1994, 1998; Barry and Schamber, 1998; Park, 1993) has shown that users making relevance judgments base their decisions on relevance criteria (document clues). These criteria are subjective and they identify various characteristics of documents as perceived by the user. Relevance criteria include the topic of the search but extend beyond that to include document characteristics such as currency of the document and bias of the author.
Evaluating a document as relevant or not relevant using simple dichotomous relevance does not provide a complete measure of the user's relevance assessment. Identifying the user's relevance criteria provides a more complete picture of the relevance assessment process and has the potential to provide a richer understanding of how users are making relevance judgments. A better understanding of how and why relevance judgments are made can provide direct suggestions for the improvement of IR systems.
Studies (Schamber et al., 1990; Bateman, 1998; Vakkari, 2000) have also shown that relevance is dynamic. Different users with different tasks assess document relevance differently and these assessments are relative to context and change over time. As users conduct a search they scan documents and their store of knowledge changes. A document containing certain information may be considered relevant or partially relevant early in the search process, but a similar document containing similar information may not be considered relevant later in the search process. A better understanding of the relevance judgment process can provide suggestions on how to make IR systems more adaptable to these cognitive changes during the search process.
This exploratory study provides empirical evidence for the dynamic and multifaceted nature of relevance, adds clarity and depth to previous studies, and extends these studies (Vakkari, 2000; Tang and Solomon, 2001; Wang and White, 1999) by identifying associations between the selection of specific relevance criteria and progression through the information search process.
This study collected and analyzed empirical data for relevance criteria selected by users during various phases of the information search process (ISP). Frequencies of relevance criteria selection choices indicate the importance of these criteria to users, and when the criteria choice frequencies are associated with stages in the search process using statistical methods, this data provides a clearer understanding of the cognitive changes occurring during the search process. Our findings provide some indication that these cognitive changes manifested as relevance criteria choices change over the course of the information search process. These changes suggest that current static IR systems could potentially benefit from a more dynamic, adaptive design, which acknowledges the cognitive changes of users.
Related research
The nature of relevance has been a complex issue throughout the history of information science. Relevance has been cited as a foundational concept of information retrieval as early as Vickery (1959). The indistinct philosophical foundations of the concept of relevance coupled with the difficulty of measuring the cognitive and situational aspects of it have led to a number of definitions of the term in information science with no single, canonical interpretation of the concept. Though the use of a dichotomous relevance as a metric prevailed in early IR research, there has always been concern that the underlying assumptions of relevance measures were flawed. Early in the history of information science Cuadra and Katter (1967) noted the consensus at the time was that separate individuals making relevance judgments often disagree and those relevance judgments by the same user might change over time. These two observations echo the assessment by Schamber et al. (1990) nearly 25 years later that relevance is both dynamic (changing over time) and multifaceted (varying among users). Saracevic (1975) distinguishes between the “system view” of relevance where the focus is on the system, and a “destination view” of relevance where the destination represents interactions with the user. It is this destination view that brings the focus of relevance to the user's interpretations and perceptions of documents being evaluated.
Schamber et al. (1990) focused on the relevance from the user's perspective but took issue with the topical bias of relevance definitions noting that relevance has been based on topicality using a best match between topic denoted in a query and the topic of the document (p. 758). They consider this approach to be problematic as it is based on the assumption that the subject terms used for the topical searching represent meanings both in the query and the document. Identifying meanings in text is an elusive exercise at best, and research indicates there is some question that the user is even aware of the meaning of their search at various stages in the search process (Cooper, 1971; Belkin, 2000). Schamber et al. (1990), note that topic‐based searches fail to capture the full breadth of the user's information need and in order to form a better understanding the dynamic and situational aspects of relevance judgments research should examine user's relevance criteria and its interactions with information seeking behavior (p. 773).
Definitions and frameworks
A number of definitions, frameworks and models exist for the concept of relevance in information science. There is currently no consensus on which is correct. The studies cited here present a theoretical basis for the definitions and frameworks used in this study.
Schamber et al. (1990) noted that relevance was a dynamic attribute of an information search. Mizzaro's (1998) theoretical work reinforces this concept and refers to this dynamic property as the “time dimension” and indicates as other researchers have that what may be relevant at one point in time may not be relevant at another point in time. In his formal model, Mizzaro sees a user in a “problematic situation” progressing through three operations: perception, expression and formalization with the result of the formalization being a query. These operations are seen as a function of time. The “stereotypes of tasks” presented identifies characteristics of documents that build on cognitive, user‐centered research. These document characteristics include but are not limited to type of document, document character (theoretical, review), page length of document and date of document (publication date, meeting date). These characteristics cited by Mizzaro mirror the relevance criteria identified by other researchers (Barry, 1994, 1998; Barry and Schamber, 1998; Park, 1993).
Other studies have explicated various attributes of relevance. Cosijn and Ingwersen (2001) built on the work of Sarcevic (1996) and others to develop a revised table of attributes and manifestations of relevance (p. 547). The manifestations of relevance identified are topical, cognitive/pertinence, situational/utility and socio‐cognitive. These are categorized as affective relevance (from Sarcevic, 1996). These affective manifestations of relevance represent expressions of cognitive changes so it is not surprising that they can be associated directly with the relevance criteria and categories identified in the user‐centered cognitive studies by Barry (1994) and others (Barry and Schamber, 1998; Park, 1993).
That these manifestations of relevance change over time has been examined in theoretical studies. Cosijn and Ingwersen (2001) emphasize that “interaction” as an attribute of relevance is dependent on time indicating that as a user progresses through a search process, affective relevance manifestations may change. Cosijn and Ingwersen note that it is the time dimension, the progression of time, “that influences the user's (relevance) decisions” and it is the cognitive changes which occur over time through interaction that leads to this influence (p. 544). Mizzaro (1998) also makes this observation.
These studies indicate that relevance is not dichotomous, nor is it a property of a document, but is instead subjective, dynamic and multidimensional. Relevance judgments vary by user and vary over time. Research should acknowledge these dynamic and situational attributes of relevance and to further our understanding of this concept, studies should investigate interactions between them (Schamber et al., 1990). To study the dynamic aspect of relevance judgments, a framework for the time dimension as described by Mizzaro (1998) is required. The concept of the time dimension as applied to IR research involves the span of time from when the user recognizes their information need to the conclusion of their efforts to resolve that information need. This span of time encompasses the information search process. Empirical research, which examines the relevance criteria selections in relation to progress through an information search process requires a search process model. Various studies have identified behavioral and process‐based models as discussed in the next section.
The information search process
In the context of this study, information behavior encompasses the activities of identifying an information need and searching for and using information sought (Wilson, 1999). What is of specific interest is how the choice of relevance criteria may change over the course of searching for information. Our interest is in the study of detailed interactions in the information‐seeking portion of information behavior and thus requires a micro‐level framework within which to perform this study.
Kuhlthau (1991, 1993) performed a series of empirical studies, which identified a series of search stages. Kuhlthau's model was built on personal construct theory, Taylor's (1968) stages of need formation and Belkin's (1982) anomalous states of knowledge and theories and models of expression and mood. The model is often interpreted as being strictly sequential, though Kuhlthau interpreted these stages as potentially being iterative and recursive (Kuhlthau, 1993, p. 69).
Ellis (1989, 1997) examined a narrower sample of users than Kuhlthau (1991) with more specific knowledge in the subject domain in which they were searching. He identified surveying, chaining, monitoring, browsing, distinguishing, filtering, extracting and ending as search patterns used for gathering information to fulfill an information need. Ellis's model is not process‐based, but behavioral. Kuhlthau's model is process‐based, but based on the observed behaviors of patrons in the library. Ellis's model emphasizes the subjective, iterative nature of the search process and avoids suggesting there is a consistent sequential process taking place. Wilson (1999) has derived a stage process from Ellis's model which suggests the “starting” pattern logically precedes other patterns and that extracting, verifying and ending represent patterns which conclude the search process (p. 255). Wilson takes this synthesis further and produces a comparison of Ellis and Kulthau's models (p. 256) which maps Ellis's behaviors into Kuhlthau's stages. While Wilson acknowledges the iterative nature of these processes, the models as drawn do not show iteration or recursion. Ellis however emphasizes the iterative non‐sequential nature of the search process more than Kuhlthau, but Kuhlthau does acknowledge that process stages may not appear in a sequential fashion and user's may move from a later stage to a preceding stage. What Kuhlthau does not address is the exclusivity of stages, an element that Ellis does address by treating these activities as behaviors. Multiple behaviors could occur simultaneously. Users may report browsing, chaining and differentiating behaviors as concurrent behaviors in Ellis's model and since they are behaviors this is appropriate. Kuhlthau's model identified activities which do not always allow for concurrent execution. For example, the case of a user reporting the activities of exploration and formulation (two separate stages in Kuhlthau's model) is not intuitive.
The ISP is used in the context of this research as a means of measuring changes in relevance criteria preferences as a function of a time dimension. The time dimension is measured using the behavioral aspects of the Kuhlthau's and Ellis's model, as synthesized by Wilson (1999). This approach provides a list of behaviors with which subjects can identify a progression through the search process, and allows an empirical observation of relevance judgments and related criteria choices in relation to progress through the search process.
Relevance criteria studies
More recent research has begun to recognize specific criteria used to make relevance judgments. Schamber (1991) conducted conducted criteria research with 30 users in three different occupational fields and identified a number of criteria used to judge relevance. Barry (1994) conducted a study which identified 23 categories of relevance criteria which applied not only to the information content of the document, but to subjective aspects of the document interpretation such as the user's beliefs and previous knowledge and to contextual factors such as other sources of information in the environment and the user's situation (situational relevance), and the quality of the source of the document (reputation, visibility, authority). A complete range of relevant, partially relevant and not relevant documents were used in the study. Barry and Schamber (1998) later combined the data collected from the Barry (1994) study with Schamber (1991). Despite the diversity of subjects' backgrounds, there was consistency in the criteria selected by the groups in the two different studies.
Barry's (1994, 1998) studies were effective in identifying a set of document attributes and contextual and situational characteristics which searchers use to assess a document as relevant or not. The criteria were categorized into the groupings identified in Table I, which represent a cross‐section of the attributes and manifestations of relevance as identified by Cosijn and Ingwersen (2001). The relevance criteria and categories reported in the study have been found other studies (Park, 1993; Maglaughlin and Sonnenwald, 2002; Tang and Solomon, 1998). The criteria identified however did conflate environmental/situational characteristics such as obtainability and cost with document characteristics such as depth,scope and recency. While this identification and categorization is consistent with Barry's exploratory research goals, it does mix the cognitive and situational aspects of relevance judgments and further relevance criteria research analysis provide a distinction between these aspects. This distinction is important for the design of this study and will be explained in more detail in the methods section.
Various studies have sought to condense and synthesize the criteria found in previous studies. Schamber and Bateman (1996) used results of three previous studies by Schamber (1991), Su (1993) and Barry (1994) and reduced and synthesized the number of relevance criteria. Maglaughlin and Sonnenwald (2002) identified 29 relevance criteria consistent with previous research (Barry, 1994; Park, 1993). Maglaughlin and Sonnenwald also found more relevance criteria in relevant documents than in non‐relevant documents, possibly indicating relevant documents are read more closely.
Summary of relevance criteria studies
As these studies illustrate, there are criteria beyond topicality which users use to evaluate whether or not a document is relevant. The study of these criteria extends back to Cuadra and Katter (1967) who studied them as intervening variables during the relevance judgment process. More recent research by Barry (1994, 1998) and others (Maglaughlin and Sonnenwald, 2002; Xu and Chen, 2006) reported here have identified a consistent set of criteria as identified in this literature review. It is important to note that Barry's studies involved all documents evaluated by users, regardless of range and direction of the relevance assessment (relevant, partially relevant, not relevant). Other studies have duplicated this methodology and have argued for the importance of evaluating partially relevant documents and negative relevance judgments (Spink et al., 1998; Hjørland, 2000). Any study of relevance criteria choices should therefore capture all levels (degrees) of relevance, from relevant, to partially relevant to not relevant.
Relevance research should examine interactions between relevance criteria and information seeking behavior (Schamber et al., 1990). Understanding these interactions can provide insight into the user's cognitive processes and identify document criteria deemed valuable in making relevance judgments from the user's perspective. The studies presented in the following section have pursued this goal to various degrees.
Relevance criteria studied in relation to the information search process
Several empirical studies have examined the relationship between relevance criteria choices and an information search process. Bateman (1998) grouped criteria identified by Barry (1994) and Schamber and Bateman (1996) into nine categories. Bateman reported that subjects did not report moving through the information search process in a “uniform manner” reporting an uneven distribution of stages with some respondents reporting being in multiple stages at once (p. 27). That this study does not report variations in criteria importance across the ISP contradicts other studies which report some criteria are more important to users in later stages (Vakkari, 2000; Vakkari and Hakala, 2000; Tang and Solomon, 2001; Wang and White, 1999; Hirsh, 1999; Taylor et al., 2007). The limited sample size and the statistical methods used may not have been sensitive enough to detect these changes. Bateman was also working with only highly relevant documents and this may have skewed the result set towards a more homogeneous set of documents and led to the exclusion of partially relevant documents that subjects elected to use or drop based on specific relevance criteria.
Spink et al. (1998) examined degrees of relevance by combining the work of four different studies on user relevance assessments during the search process. The studies involved 55 searchers conducting initial searches to solve an information problem. Researchers found an association between the number of partially relevant items retrieved and reported changes in a user's personal knowledge about the research problem. Partially relevant documents appeared to be more important in the early stages. Based on this research, it would appear that partially relevant documents may contain relevance criteria that user's evaluate to determine whether or not the document is valid.
Tang and Solomon (1998) used a naturalistic approach to study the search behavior of a single user preparing a term paper. They considered the information search process (time dimension) in their study as two stages of document evaluation. The authors report that the subject appeared to be more certain of the search (as the subject's mental model changed) later in the search process. The authors also report some “fuzziness” of relevance observations during the process, confirming the need for partial relevance judgments.
Tang and Solomon (2001) conducted additional relevance criteria studies using both laboratory and naturalistic approaches. The laboratory experiment results showed a change in use of relevance criteria choices moving from the two stages for the criteria of clarity, importance, newness, recency, topical focus, topical relatedness. The study limited evaluation to only two stages of the ISP, so additional information on other ISP stages/activities was not collected.
Wang and Soergel (1998) reported on a longitudinal study of 25 students and faculty filling an information need as part of a research effort. The authors note the “epistemic value” of a document as the prerequisite for all other values for the document (see also Hjørland, 2000). This is logical given the sample population and the search task, as the author notes they were working with “experts” in the field (p. 130). The methodology used collected both relevant and non‐relevant document judgments. The resulting criteria are consistent with those found in previous research and include discipline, quality and novelty. Topicality was rated highest, followed by orientation/level. The criteria of “ability to understand” or “comprehensibility” is missing from their list of criteria. This may be due to the expertise level of their subjects who as experts were able to comprehend the content of all documents reviewed. The researchers did not report changes in relevance criteria over time.
Crystal and Greenberg (2006) performed a study using 12 subjects who examined documents found on the worldwide web and identified relevance criteria in the document surrogate and the document. Using content analysis and statistical analysis they identified a number of relevance criteria and found that a few relevance criteria were commonly identified by subjects and a larger number were identified much less frequently. The criteria of “topicality” and “research group” were criteria frequently identified by their subjects, consistent with the suggestion by Wang and Soergel (1998) that epistemic value (research group) must be satisfied before other search criteria are considered in the search process.
Wang and White (1999) conducted a long‐term study of a convenience sample of 15 experienced researchers with the pool being composed of eight professors, six doctoral students and one masters student. Stages in the ISP were identified as “selecting”, “reading” and “citing”. Researchers identified topicality, novelty and recency as the most commonly selected relevance criteria and added a previously unreported topic of “cognitive requisite” (the ability to comprehend a document). The subjects tended to select more documents as relevant than they actually used and subjects applied more diverse relevance criteria in later stages with the “citing” stage showing the greatest variety of selection criteria. The study adds evidence for the dynamic nature of the search process, but does not report specific relevance criteria preferences in the ISP stages.
Choi and Rasmussen (2002) examined the use of relevance criteria information in making relevance judgments on image selection. Subjects were 18 faculty members and 20 graduate students. The common criteria chosen were topicality, lack of textual information for the image, lack of clarity for the image and lack of novelty. These results for graphical images are similar to the criteria reported for text documents and add to the evidence that topicality is a core criteria along with novelty and clarity. The limited number of ISP stages used limits the ability to determine associations between relevance criteria chosen and ISP stage.
Hirsh (1999) performed a study with ten fifth grade children (ages ten to 11) as they performed research for an assigned project. Many of the categories identified by other researchers for adults re‐occur with this age group with different levels of importance, e.g. topicality, recency, novelty. Factors such as peer‐interest are more commonly reported and convenience/accessibility also appears to be important to the subjects. The authors examined the time dimension using Kuhlthau's (1991) model though activity at specific stages was not reported. The study adds to the evidence that the use of topicality decreases in later search stages as users begin evaluating documents on a wider range of relevance criteria. The age group of the subjects, however, limits comparability to other studies with adult subjects.
Vakkari (2000) and Vakkari and Hakala (2000) examined changes in relevance criteria and task performance and reports on a study which involved eleven students preparing a proposal for a master's thesis. Results from these studies, suggests that, users identified more relevant documents early in the search process and identified fewer relevant documents later in the search process. The authors speculated that this change in relevance assessment might be due to the users store of knowledge improving during the search process and the user being more discerning in relevance judgments as their store of knowledge improves. The authors also identify some changes in relevance criteria as users progress through the search process, with users more likely to choose “novelty” as an important category later in the search process. They speculate that users having selected a set of relevant documents earlier in the search process are more interested in finding novel information (documents different than their current selected set) later in the search process.
Vakkari (2000) used several stages of the the Kuhlthau ISP model to track the progress of his subjects. The researcher found that subjects moved through the search process at different paces often reporting that they were in multiple stages in tandem. Researchers reported that the categories of “novelty” and “interest” were selected more during the later stages of the ISP and “topicality” was the most commonly selected criteria. Though this study provided some useful insights, its conclusions were based on descriptive statistics using a small sample size, which limit their statistical strength.
Taylor et al. (2007) performed a study in which researchers identified criteria used for relevance judgments by performing content analysis of comments made by subjects during document selection. A random sample of 40 subjects from the results of a previous study with 300 undergraduate students was used. Researchers used four search stages consistent with Kuhlthau (1993). Findings suggested criteria selection changes, by subjects, as they progressed through a search for documents relevant to an information need. The authors report a statistically significant relationship in terms of frequency of selection for the preference of certain criteria in early search stages (”recency” and “specificity”) and for other criteria in later search stages (”source novelty” and “interest”).
Summary: literature review
While these studies did examine the use of various criteria to determine relevance, few studies examined these relevance criteria choices in relation to the search process and those that did provided very little detail. These studies were consistent in their use of content analysis and descriptive statistics to identify relevance criteria. Researchers identified similar relevance criteria across all studies using various problem domains indicating that user's consistently use similar relevance criteria to make relevance judgments.
Most studies used small subject pools and were thus limited to descriptive statistics for analysis. Use of larger subject pools and stronger statistical methods will add to our understanding of the use of relevance criteria during the search process. The identification of statistically significant criteria will extend previous findings of association with the identification of associations between specific criteria and the information search process.
Research questions
In this paper, relevance is defined as the user's perceptions of the document's importance to their information need and is both multidimensional (varying among users) and dynamic (varying over time). This concept of relevance has been embraced by a number of researchers and provides a “real world” view of relevance suitable for information science research (Schamber et al., 1990; Wilson, 1973; Harter, 1992; Sarcevic, 1996; Borlund, 2003; Borlund and Ingwersen, 1998). For purposes of this study, relevance judgments are defined as the process of a user evaluating a document or document representation as being relevant or not relevant to their information need using a categorical scale. Relevance criteria are those factors that contribute to the user's relevance assessment for both a positive (document is relevant) and negative (document is not relevant) assessment.
This study examined the use of relevance criteria by the searcher during the ISP, looking for changes in relevance criteria deemed important to the user as they progress through the ISP. We also examined the potential influences of some situational and contextual factors, which might affect relevance judgments.
The following research questions were examined in this study.
- 1.
Which relevance criteria are most commonly used during which stages of the IR process?
- 2.
What is the nature of the relationship between the stage of the ISP reported and the relevance criteria considered important by the user?
Research design
The research design involved a short duration study of subjects conducting searches in a laboratory where they could be monitored. Subjects worked alone in this environment to solve an assigned research problem. This design was chosen to avoid the overhead and lack of control in a long‐term (longitudinal) study were subjects would work in an unmonitored environment. It was felt that the unmonitored, long‐duration approach would lead to haphazard reporting and questionable results. In a monitored environment over a short period of time, subjects would be more focused and data collected would be of a higher quality.
The research protocol involved the collection of data through three sets of surveys: pre‐test, in‐test and post‐test. The pre‐test survey asked the subject to provide general information about their background and their knowledge of the problem domain. The in‐test survey asked the subject specific questions about the document being reviewed by the subject. The post‐test survey asked questions concerning the subject's general satisfaction with the search process. Surveys were pre‐tested during a pilot study the results of which suggested minor modifications.
Subjects for this study were a convenience sample from a pool of undergraduate students at an American university. Subjects volunteered for the research. All subjects who volunteered were directed to report to a computer laboratory at an assigned time to participate in the research. Once in the laboratory, subjects were assigned to a computer workstation and given the research question shown in the following. All subjects were assigned the same research question:
Consider that you have been assigned the following question as part of an open book, open internet exam. Conduct a search for documents, which you could find useful in answering these questions. Attempt to find at least ten documents which you could find useful.Compare and contrast the benefits of using a fixed exchange rate versus a flexible exchange rate for international transactions.
Subjects worked alone to gather information to answer the research question. Subjects used an online information search service that allowed them to find journal articles on their assigned subject. Subjects could review document representations (descriptions of the journal article), or could review the actual journal article.
Subjects were told to record document relevance, search stage, and criteria used to judge document relevance for each document reviewed. Subjects recorded this information about their search using online data collection instruments as detailed in the “Data collection” section below.
Search process
The ISP stage references (see Table II) used in this study were a combination of Kuhlthau's (1993) ISP stages and Ellis's (1997) search patterns. The first two ISP stages of Kuhlthau (Task Initiation and Topic Selection) were not used since the search task and topic for this study were provided as part of the methodology and have essentially been completed for the subjects. The ISP stages developed by Kuhlthau were based on research conducted using mediated library searches. The process of online searching as conducted in this study does not involve the use of mediators and involves activities that were not covered in detail by Kuhlthau's studies. To address this issue, Ellis patterns of browsing and extracting were used to provide additional depth and are search behaviors more amenable to the online searching used in this study. The activities identified here were not treated as sequential or mutually exclusive activities. Subjects could select more than one activity and were not required to use any particular order. This is consistent with Ellis (1989) who identified search behavior patterns and did not observe a sequential order in the selection of those patterns. Selection of search stages were therefore aligned with Ellis's (1997) concept of information behavior patterns and Kuhlthau's (1991) notion that stages could be repeated. It is unclear how well these models apply to search behavior on the web, though it is reasonable to assume that the same search behaviors that took place in mediated searches in the library (as in Kuhlthau's studies), or in the behavioral study performed by Ellis (1997) would continue with online searches. The search stages and descriptions presented to subjects are shown in Table III. Subjects were allowed to select multiple search stages in parallel, were not required to select the stages in any particular sequence.
Relevance criteria
Subjects indicated document relevance on a scale of 1 to 10, with 1 being least relevant and 10 being most relevant (Appendix 1). The number of criteria was reduced for clarity and to avoid survey exhaustion on the part of the subjects.
Data collection process
All subjects were assigned the same specific search task (see Appendix 2) and performed the task while being monitored in a computer lab at the university. Online, web‐based surveys were used to collect data on the subject's background, their relevance criteria choices, relevance assessments and ISP stage. All data was self‐reported by the subject. Subjects worked alone on their search problem and took between 45 minutes and two hours to complete the test session.
The information search was conducted with online searches using the ABI/Inform online database to find documents in that database to help solve their search problem. The ABI/Inform database which indexes a combination of trade journals and scholarly journals using a search engine interface similar to Google. Document selections were printed and reviewed by the subjects in the lab during the session. After reviewing a set of ten documents, subjects were asked to complete a survey on each document explaining why it was relevant or not relevant and identifying which relevance categories were useful in making that assessment. All surveys were completed online by the subjects during the session.
In order to encourage subjects to perform searches, collect documents and review them, subjects were asked to retrieve at least ten documents to solve an assigned research question. To hold the search task constant in this study, a single research question was assigned to all subject (see Appendix 2). Documents retrieved could be any combination of “relevant” and “not relevant” documents. The relevance assessment questions, search stage questions and search question used in the study are listed in the Appendices. For each document selected, subjects recorded the relevance criteria, which they used to assess the document, the search stage they were in when they evaluated the document and whether or not they considered the document relevant to answering the search question they had been assigned. Relevance was captured on an interval scale and allowed for a full range of relevance including partial relevance.
On completion of the search test, a user interview was conducted using open‐ended questions about the documents selected and the subject's reasons for considering the document relevant or not relevant. These subject interviews were recorded and later transcribed.
The subject's domain knowledge was assessed through an objective test on subject area knowledge. This score was then converted into a binary variable indicating the subject's knowledge of the domain. based on the descriptive statistics used to evaluate the score (n=383, M=72, SD=14) a score of 80 percent or higher was interpreted as above average and was used to indicate a knowledgeable user.
Results of data collection
A convenience sample of 39 subjects was drawn from the student population of business school students at an American university. The subject pool was a mix of approximately 20 percent paid subjects and 80 percent unpaid subjects.
The data collected from the research sessions consisted of 383 records each representing a relevant document that a subject used in his/her search process, and corresponding relevance criteria and ISP stage selections. Documents could be used in more than one stage of the search process, and some subjects did report on the same document in more than one stage. When each relevance judgment was joined with a search stage, the 383 rows of relevance judgments generated 795 separate judgment‐stage reports. Subjects were not required to report particular ISP stages and were not required to report ISP stages in a specific order. An examination of the percentage of relevance criteria category selections as a percentage of total selections in Table IV indicates no particular criteria dominated the users' selection of criteria. The results shown in Table V shows that with the exception of the “presentation” search stage, search stages were selected somewhat consistently, with subjects showing a slight preference for the “focusing,” “browsing,” and “extracting” search stage over other search stages. Because of the low selection counts for the “presentation” stage, and the difficulty of using some statistical analysis methods with low counts, the results for the selection of the “presentation” search stage were not used in the analysis.
Statistical analysis of cross‐tabulations
The following section contains cross tabulations of relevance criteria choices and information search process stage. An analysis of variance test was used to examine the values in these tables to determine if a relationship exists between the column values (the categories) and the values in the rows (the cases). These statistical tests determine the probability that the variables represented in the row and column are related, with the value of p representing the probability that the relationship is due to chance and not a valid effect. A lower p value indicates that the relationship between the two values is more likely due to a relationship. When cell values are low, the statistical strength of analysis of variance tests is reduced. With Chi square tests, a Yates correction can be applied to improve the validity of results with low cell values in the results table. The Yates correction was applied to the calculations performed in this section to increase the validity of the results when the frequency counts in cells was low.
Relevance judgment groups
The research methods for this study had subjects indicate the degree of relevance when they made a relevance judgment. The degree of relevance was selected as a value between 1 and 10, with 1 being the least relevant and 10 being the most relevant. As part of data filtering process, these scores were folded into three distinctive bands of relevance: low relevance interpreted as a value of 3 or less assigned to the degree of relevance weight; high relevance interpreted as a value of 8 or greater assigned to the degree of relevance weight; and partial relevance was interpreted as a value between 4 and 7 assigned to the degree of relevance weight.
Partial relevance
Analysis involved looking for statistically significant variance in the frequency of selection across search stages. Table VI shows frequency counts for partial (a value between 4 and 7) relevance judgments by search stage in relation to relevance criteria used. Examination of these results using Chi square did not reveal statistically significant relationships.
High relevance judgments
Table VII contains a cross‐tabulation of the criteria selections across all search stages for high relevance document selections. Taken as a whole, these results do not demonstrate a statistically significant association between relevance criteria selection and search stage. Additional analysis, however, identified other associations.
The search stage counts in Table VII show that subjects were more likely to select some search stages over others, a tendency which could skew analysis of variance results. Table V and related Figure 1 examine the subset of criteria, which varies in frequency of selection across search stages. Figure 1 shows that in terms of frequency of selection for high relevance, there is a shift in the importance of some criteria across search stages. Most selections appear to peak during the “browsing” stage, and an examination of the total row in Table VII indicates that this is the most commonly selected search stage. Since subjects appeared to have a preference for that search stage in general (as shown in Table VII), thus increasing the possibility that selection of a particular criteria in the “browsing” stage may have had more to do with a subject's preference for the “browsing” stage in general, than a preference for a particular relevance criteria in a particular search stage. Analysis of the relationship between criteria selections in other search stages would provide a stronger indication that search stage impacts criteria selection.
To provide greater evidence for the selectivity of criteria choices varying over search stage, an additional data filtering mechanism was applied by selecting search stages which are approximately equal in frequency of document selection: “learning,” “browsing,” “extracting,” and “focusing.” With these stage selections being roughly equal, variations in category selections between search stages are more likely to be due to preference for relevance criteria than for search stage.
The Chi square analysis of variance formula was applied to the results shown in Table VIII. Analysis of the data determined that there was no statistical relationship when a Chi Square test was performed across all search stages, but an examination of the relationship between specific search stages revealed a statistical relationship between the “focusing” and “browsing” stage for the criteria reported in Table IX. These results indicate that for this sample, the criteria of “ability to understand”, “clarity” and “recency” are selected more in the “browsing” stage than in the “focusing” stage, and this frequency of selection has the statistical significance shown in Table IX.
Low relevance judgments
Table X shows the frequency counts for relevance criteria selections with low relevance (a selection of three or less on a scale of 1 to 10) across search stages. Taken as a whole, these results do not produce a statistically significant relationship. Additional analysis was performed using specific search stages with similar levels of selection by subjects did detect relationships.
The selection of more low relevance criteria selections as subjects progress through the search process would suggest a decrease in importance for those criteria as measured by frequency of selection. Conversely, a selection of fewer low relevance criteria would suggest that the criteria is increasing in significance as subjects progress through the search process. Table XI identifies those criteria‐stage results, which appear to vary across search stages. Figure 2 provides a graphical representation of these data. Various relationships were analyzed, and Table XII provides a list of those statistical relationships. Note that criteria such as “ability to understand”, “instructional”, “precision”, and “recency” appear to have fewer low relevance criteria selections as subjects progress through the search process, possibly indicating these criteria are increasing in importance to subjects.
As shown in Table XII, there were statistically significant associations for the “ability to understand” and the “amount of information” criteria moving from the “learning” stage to the “focusing” stage. These statistical results indicate subjects were more likely to select criteria using a low rating in the “focusing” stage than in the “learning” stage.
Conversely, in evaluating the progress from the focusing stage to the extracting stage, results are found which indicate a statistically significant decrease in the selection of criteria using a low relevance ranking. As shown in Table XIII, the criteria of “recency,” “instructional,” and “authority” all showed a decrease in selection, an indication that these criteria may have increased in significance (since fewer lower relevant selections were made).
Single search stage per relevance selection analysis
In this study subjects were allowed to select more than one search stage for each relevance judgment. As shown in Table XIV, a number of subjects reported being in more than one search stage as they judged document relevance. This could indicate that some subjects were confused by the search stage selection and did not discriminate in the selection of search stage, or it could represent a valid reporting of search stage behavior in lieu of a search stage (as in Ellis, 1997). To strengthen these results, analysis was done with subjects who selected one search stage per relevance judgment.
The frequency counts for relevance criteria selections for single search stage selections for relevance importance values greater than 5 is shown in Table XV. These results are shown graphically in Figure 3, which indicates the variation in selection is fairly consistent for most criteria used. However, for the criteria of “clarity,” “recency,” “depth,” and “scope” there is statistically significant variation in moving from the “focusing” stage to the “extracting” search stage as shown in Table XVI, an indication these criteria were more important to subjects in later ISP stages.
Comparison of results
Table XVII compares the results of relationships of high relevance, low relevance and single search stage selections. The criteria of “recency”, “instructional”, and “authority” demonstrated statistical variations at various levels of significance in either the high and low relevance analysis, strengthening the case for the underlying relationship between relevance criteria selection and search stage. The criteria of “recency” also demonstrated statistically significant results in the analysis of single search stage, further strengthening the case for the relationship of search stage and relevance criteria selections.
Discussion
The data collected in this study provide empirical evidence of the relationship of relevance judgment choices to relevance judgment, and of the relationship of relevance criteria choices to search stage. Statistically significant results demonstrate the relationship of relevance criteria choices to search stage, indicating that as subjects progress through the search process, their relevance criteria choices may change, with some criteria being more important in some search stages than other criteria. These changes occur as subjects progress through an individual search episode and evaluate documents and make relevance judgments.
Major findings
Specifically, the major findings based on these results are as follows.
- 1.
Subjects demonstrated a preference for the criteria of “ability to understand”, “clarity”, “depth/scope”, “precision”, and “specificity” for relevant documents over all search stages as shown in Table IV.
- 2.
Analysis found the statistical relationships shown in Table XVII, providing indication that these specific criteria increased in importance to searchers as they progressed through the information search process.
Detailed discussion
Based on these statistically significant results, it appears that based on frequency of selection, the criteria of “recency”, “instructional” increase in importance as the user progresses through the search process. This tendency was demonstrated with both the low relevance and high relevance analysis. It is possible that since the question posed to the subjects involved a topic somewhat unfamiliar with the subjects, the criteria of “instructional” was important. The selection of “authority” of the source as a criteria with low relevance also provides some indication that the subjects were evaluating who was providing the information in making their relevance decision. The criteria of “recency” also appears to be a consistently selected criteria for subjects in this study, and increases in importance as subjects progress through the ISP, even though the question did not require current information.
Subjects were assigned a question which asked them to “compare and contrast” two concepts (see Appendix 2). This should have led to some use of “bias” as part of the criteria for relevance, but though selection of that criteria selection does fluctuate over several search stages, this variation was not statistically significant. The criteria of “clarity,” “ability to understand”, “depth/scope”, and “precision” also increase in importance as subjects progress through the search process. Subjects appear to be more concerned with being able to understand and interpret the material and to find documents with “depth/scope” in later stages, and this study finds this tendency to be statistically significant.
Limitations
This was intended as an exploratory study. The number of participants adds strength to the statistical analysis, but the generalizability is limited by the purposive sample of college age subjects. The search task was intended to be outside of the subject's knowledge domain, thus requiring them to search aggressively for information. A more knowledgeable subject may search differently using different criteria and proceeding through the ISP in a different manner.
Though the search model used is considered appropriate for this study, the application of information search process models to online searching is not settled. Results in this study do suggest a uniform selection of the search stages available, suggesting the subjects did proceed through some portion of the search stages in a uniform manner (see Table V).
The convenience sample of search subjects used in this sample may have biased the results to some extent. The subjects were undergraduate students from a business school. They were not experienced researchers such as the graduate students and professionals used in several of the studies referenced in this paper. The scores on the domain knowledge assessment, a test intended to determine the subject's knowledge in the domain where they were required to search, partially confirm a bias toward a user with limited knowledge in the subject area. The average score on this assessment was 70 percent, so using a standard interpretation of test scores, fully 50 percent of the subjects had a deficient knowledge of the subject area with a domain knowledge assessment score below 70 percent.
Conclusion
These results provide further indications of the complex and dynamic nature of relevance. Statistical results indicate not only that a user's relevance assessments change during the course of the ISP, but that the changes taking place are measurable. Our results differed from those of some previous studies and were consistent with others. We feel that these differences are partially attributable to contextual factors in the search process as reflected through choices in methodology, specifically the selection of subject samples (undergraduate versus graduate), the background of these subjects and their experience and accumulated knowledge in the domain, and the structure and duration of the search test (long duration versus short duration). Future studies should try to better isolate and study these factors and their specific effect on the relevance criteria choices.
The implications of these findings provide direct suggestions for improvement in the development of information retrieval (IR) systems. While it has been suggested in the past that IR systems should be more flexible and adaptive to the iterative nature of the ISP, there have been few specific suggestions for how this could be done and what form these adaptations should be. These findings suggest that an IR system which provides document representations (indices), which correspond to relevance categories (i.e. authoritative, ability to teach, depth/scope) could lead to improved searches. Instead of browsing to find documents, which match a specific relevance criteria, the searcher could select the relevance criteria as part of the search criteria. As the search process progresses, the IR system could adapt to provide additional suggestions for those criteria which appear to be more important later in the search process.
While the ISP model used in this study was adequate, the description of search stages may have been confusing to the user. Table V reveals that subjects most commonly selected “browsing for information” (17.25 percent versus a mean of 12.5 percent) and “extracting useful information” (16.23 percent versus a mean of 12.5 percent) as search stage choices. These are commonly understood terms for “searching” on the Web and are concepts that may have biased subject's choices of search stages. A more refined model with a finer granularity of choices may provide users with more useful choices.
Appendix 1. Relevance questions
0 What was your reason for selecting/rejecting this document?
1 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Amount of information”
2 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Specificity”
3 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Clarity of presentation”
4 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Ability to Understand”
5 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Depth/Scope”
6 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Precision of Document”
7 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Recency of Document Publication”
9 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Interest in Topic”
10 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Instructional”
11 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Authority of Author”
12 On a scale of 1 to 10 indicate why you considered this document relevant/irrelevant based on the criteria of “Bias of Author”
Appendix 2. Search question
Consider that you have been assigned the following question as part of an open book, open Internet exam. Conduct a search for documents, which you would find useful in answering these questions. Attempt to find at least ten documents which you would find useful.
Compare and contrast the benefits of using a fixed exchange rate versus a flexible exchange rate for international transactions.
Appendix 3. Search stage questions
2. Which of the following applies to where you were in your search process when you accepted or rejected this document.
□a.) Becoming informed on the topic
□b.) Learning about the topic
□c.) Trying to focus on the topic/subtopic
□d.) Defining and extending focus
□e.) Browsing for information on the focus I've identified
□f.) Extracting useful information
□g.) Verifying information retrieved
□h.) Completion and presentation of information
Appendix 4. Pre‐test questionnaire
1 How often do you perform searches using online information databases (1=not very often; 7=very often)?
2 When you search for information, which of the following do you usually use?
3 How frequently do you perform online searches?
4 At this point in time what is your highest level of education?
5 Rate your experience level with online search engines (1=lowest, 7=highest).
6 How would you rate your expertise in searching for information using online databases (1=least, 7=most)?
7 How familiar are you with the ABI/Inform database and search engine (1=least, 7=most)?
8 How would you rate your expertise in using the Google online search engine (1=least, 7=most)?
9 Do you commonly use advanced search engine features?
Appendix 5. Post‐test questionnaire
1 How useful do you think the search process has been (1=not very useful; 7=very useful)?
2 How effective was the search process (1=not very effective; 7=very effective)?
3 Indicate your level of satisfaction with the information/documents you found during this search (1=not very satisfied; 7=very satisfied).
4 Do you feel able to answer the question and fulfill the requirements of the search assignment with the information retrieved?
5 Is there anything you would like to suggest to further improve the search process?
Corresponding author
Art Taylor can be contacted at: [email protected]
References
Barry, C.L. (1998), “Document representations and clues to document relevance”, Journal of the American Society for Information Science, Vol. 49 No. 14, pp. 1293‐303.
Barry, C.L. and Schamber, L. (1998), “Users' criteria for relevance evaluation: a cross‐situational comparison”, Information Processing and Management, Vol. 34 Nos 2‐3, pp. 219‐36.
Barry, C.L. (1994), “User‐defined relevance criteria: an exploratory study”, Journal of the American Society for Information Science, Vol. 45, pp. 149‐59.
Bateman, J. (1998), “Changes in relevance criteria: a longitudinal study”, Proceedings of the ASIS Annual Meeting, October 25‐29, Pittsburgh, PA, Vol. 35, pp. 23‐32.
Belkin, N. (1982), “ASK for information retrieval – part I”, Journal of Documentation, Vol. 38, pp. 61‐72.
Belkin, N.J. (2000), “Helping people find what they don't know”, Communications of ACM, Vol. 43, pp. 58‐61.
Borlund, P. (2003), “The concept of relevance in IR”, Journal of the American Society for Information Science and Technology, Vol. 54, pp. 913‐25.
Borlund, P. and Ingwersen, P. (1998), “Measures of relative relevance and ranked half‐life: performance indicators for interactive IR”, SIGIR '98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24‐28, Melbourne, Australia, ACM, pp. 324‐31.
Choi, Y. and Rasmussen, E.M. (2002), “Users' relevance criteria in image retrieval in American history”, Information Processing & Management, Vol. 38, pp. 695‐726.
Cooper, W.S. (1971), “A definition of relevance for information retrieval”, Information Storage and Retrieval, Vol. 7, pp. 19‐37.
Cosijn, E. and Ingwersen, P. (2001), “Dimensions of relevance”, Information Processing and Management, Vol. 36 No. 4, pp. 533‐50.
Crystal, A. and Greenberg, J. (2006), “Relevance criteria identified by health information users during web searches”, Journal of the American Society for Information Science and Technology, Vol. 57 No. 10, pp. 1368‐82.
Cuadra, C.A. and Katter, R.V. (1967), “Opening the black box of ‘relevance’”, Journal of Documentation, Vol. 23, pp. 291‐303.
Ellis, D. (1989), “A behavioural approach to information science retrieval design”, Journal of Documentation, Vol. 45, pp. 171‐212.
Ellis, D. (1997), “Modeling the information seeking patterns of engineers and research scientists in an industrial environment”, Journal of Documentation, Vol. 53 No. 4, pp. 384‐403.
Harter, S.P. (1992), “Psychological relevance and information science”, Journal of the American Society for Information Science, Vol. 43 No. 9, pp. 602‐15.
Hirsh, S.G. (1999), “Children's relevance criteria and information seeking on electronic resources”, Journal of the American Society for Information Science, Vol. 50 No. 14, pp. 1265‐83.
Hjørland, B. (2000), “Relevance research: the missing perspective(s): non‐relevance and epistemological relevance”, Journal of the American Society for Information Science, Vol. 51, pp. 209‐11.
Kuhlthau, C. (1993), Seeking Meaning: A Process Approach to Library and Information Services, Ablex Publishing Company, Norwood, NJ.
Kuhlthau, C.C. (1991), “Inside the search process: information seeking from the user's perspective”, Journal of the American Society for Information Science, Vol. 42 No. 5, pp. 361‐71.
Maglaughlin, K.L. and Sonnenwald, D.H. (2002), “User perspectives on relevance criteria: a comparison among relevant, partially relevant, and not‐relevant judgments”, Journal of the American Society of Information Science and Technology, Vol. 53 No. 5, pp. 327‐42.
Mizzaro, S. (1998), “How many relevances in information retrieval?”, Interacting with Computers, Vol. 10, pp. 303‐20.
Park, K. (1993), “The nature of relevance in information retrieval: an empirical study”, Library Quarterly, Vol. 63 No. 3, pp. 318‐51.
Saracevic, T. (1975), “Relevance: a review of and a framework for the thinking on the notion in information science”, Journal of the American Society for Information Science, Vol. 26, pp. 321‐43.
Sarcevic, T. (1996), “Relevance reconsidered”, Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS 2), October 14‐17, Copenhagen, pp. 201‐18.
Schamber, L. (1991), “Users' criteria for evaluation in a multimedia environment”, in Griffiths, J.‐M. (Ed.), Proceedings of the American Society for Information Science (ASIS) 54th Annual Meeting, Vol. 28, pp. 126‐33.
Schamber, L. and Bateman, J. (1996), “User criteria in relevance evaluation: Toward development of a measurement scale”, Proceedings of the 59th Annual Meeting of the American Society for Information Science, Baltimore, MD, October 21‐26.
Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), “A re‐examination of relevance: toward a dynamic, situational definition”, Information Processing and Management, Vol. 26 No. 6, pp. 755‐76.
Spink, A., Greisdorf, H. and Bateman, J. (1998), “From highly relevant to not relevant: examining different regions of relevance”, Information Processing and Management, Vol. 34 No. 5, pp. 599‐621.
Su, L.T. (1993), “Is relevance an adequate criterion for retrieval system evaluation: an empirical inquiry into user's evaluation”, Proceedings of the American Society for Information Science, Learned Information, Medford, NJ, pp. 93‐103.
Tang, R. and Solomon, P. (1998), “Toward an understanding of the dynamics of relevance judgment: an analysis of one person's search behavior”, Information Processing and Management, Nos 2‐3, pp. 237‐56.
Tang, R.S. and Solomon, P. (2001), “Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies”, Journal of the American Society for Information Science and Technology, Vol. 52 No. 8, pp. 676‐85.
Taylor, A.R., Cool, C., Belkin, N.J. and Amadio, W.J. (2007), “Relationships between categories of relevance criteria and stage in task completion”, Information Processing and Management, Vol. 43, pp. 1071‐84.
Taylor, R.S. (1968), “Question‐negotiation and information seeking in libraries”, College Research Libraries, Vol. 29 No. 3, pp. 178‐94.
Vakkari, P. (2000), “Relevance and contributing information types of searched documents in task performance”, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24‐28, Athens, Greece, ACM Press, New York, NY, pp. 2‐9.
Vakkari, P. and Hakala, N. (2000), “Changes in relevance criteria and problem stages in task performance”, Journal of Documentation, Vol. 56 No. 5, pp. 540‐62.
Wang, P. and Soergel, D. (1998), “A cognitive model of document use during a research project. Study I: document selection”, Journal of the American Society of Information Science and Technology, Vol. 49 No. 2, pp. 115‐33.
Wang, P. and White, M.D. (1999), “A cognitive model of document use during a research project. Study II: decisions at the reading and writing stages”, Journal of the American Society for Information Science, Vol. 50 No. 2, pp. 98‐114.
Wilson, P. (1973), “Situational relevance”, Information Storage and Retrieval, Vol. 9 No. 8, pp. 457‐71.
Wilson, T.D. (1999), “Models in information behavior research”, Journal of Documentation, Vol. 55, pp. 49‐270.
Xu, Y. and Chen, Z. (2006), “Relevance judgment: what do information consumers consider beyond topicality?”, Journal of the American Society for Information Science and Technology, Vol. 57 No. 7, pp. 961‐73.
Further Reading
Belkin, N.J., Oddy, R.N. and Brooks, H. (1982), “ASK for information retrieval – part II”, Journal of Documentation, Vol. 38 No. 3, pp. 145‐64.
Bookstein, A. (1979), “Relevance”, Journal of the American Society for Information Science, Vol. 30 No. 5, pp. 269‐73.
Swanson, D.R. (1986), “Subjective versus objective relevance in bibliographic retrieval systems”, The Library Quarterly, Vol. 56, pp. 389‐98.
Swanson, D.R. (1988), “Historical note: information retrieval and the future of an illusion”, Journal of the American Society for Information Science, Vol. 39, pp. 92‐8.
Vakkari, P., Pennanen, M. and Serola, S. (2003), “Changes in search terms and tactics while writing a research proposal: a longitudinal case study”, Information Processing and Management, Vol. 39, pp. 445‐63.
Wilson, T.D. (1997), “Information behavior: an interdisciplinary perspective”, Information Processing and Management, Vol. 83 No. 4, pp. 551‐72.