@prefix vivo: . @prefix edm: . @prefix ns0: . @prefix dcterms: . @prefix skos: . vivo:departmentOrSchool "Arts, Faculty of"@en, "Library, Archival and Information Studies (SLAIS), School of"@en ; edm:dataProvider "DSpace"@en ; ns0:degreeCampus "UBCV"@en ; dcterms:creator "Kessler, Kristof"@en ; dcterms:issued "2013-08-30T14:45:46Z"@en, "2013"@en ; vivo:relatedDegree "Master of Library and Information Studies - MLIS"@en ; ns0:degreeGrantor "University of British Columbia"@en ; dcterms:description "As noted by Herbert Simon, the challenge presented by the rich information ecologies of our time is one of shortage of attention. Hence it is necessary for important information to stand out. This research proposes that facets used in the context of full text search, support this ‘attention getting’. Faceted search has proven to provide more effective information-seeking support to users in some situations. To date, studies have focused on specific domains typically using a specific set of facets. Consequently, little is known about the effect of faceted search on a broader range of task types. This research investigates the effect of faceted search in a task context. In this process questions about the differences in perceived usefulness and actual use, and whether systems providing facets lead to a higher user satisfaction, effectiveness, and efficiency compared to systems without this capability are answered by means of a systems review, an online questionnaire, and an experimental user study. The systems review revealed 47 potential facets used across the 12 systems perused. 14 of these facets from different levels of observed prevalence were used in the online questionnaire to determine their perceived usefulness across three types of search tasks: Doing, Known-Item, and Learning. Results of the questionnaire research show a significant difference in the perceived usefulness of the facets between Doing and Learning tasks. Six out of the 14 facets, 4 perceived as highly and 2 as less useful, were incorporated into an experimental government search system for comparison to a baseline system not providing facet capabilities. An experimental user study employing these systems found that there were some differences in the perceived usefulness and actual use of facets. Specifically, the audience facet, which received low usefulness scores in the questionnaire, was used quite frequently in the user study. Only few statistically significant differences between the baseline and experimental system were found. The most notable differences were found in Perceived Success, a measure of effectiveness, and Level of Satisfaction, a measure of satisfaction, between the first and third tasks performed in the experimental system, with the third task showing higher scores."@en ; edm:aggregatedCHO "https://circle.library.ubc.ca/rest/handle/2429/44960?expand=metadata"@en ; skos:note " TASK-TYPE-SPECIFIC USE OF FACETS IN DISCOVERING ONLINE CONTENT by KRISTOF KESSLER B.A., The Open University, 2005 M.Sc., The University of Manchester, 2010 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF LIBRARY AND INFORMATION STUDIES in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2013 ? Kristof Kessler, 2013 ii Abstract As noted by Herbert Simon, the challenge presented by the rich information ecologies of our time is one of shortage of attention. Hence it is necessary for important information to stand out. This research proposes that facets used in the context of full text search, support this ?attention getting?. Faceted search has proven to provide more effective information-seeking support to users in some situations. To date, studies have focused on specific domains typically using a specific set of facets. Consequently, little is known about the effect of faceted search on a broader range of task types. This research investigates the effect of faceted search in a task context. In this process questions about the differences in perceived usefulness and actual use, and whether systems providing facets lead to a higher user satisfaction, effectiveness, and efficiency compared to systems without this capability are answered by means of a systems review, an online questionnaire, and an experimental user study. The systems review revealed 47 potential facets used across the 12 systems perused. 14 of these facets from different levels of observed prevalence were used in the online questionnaire to determine their perceived usefulness across three types of search tasks: Doing, Known-Item, and Learning. Results of the questionnaire research show a significant difference in the perceived usefulness of the facets between Doing and Learning tasks. Six out of the 14 facets, 4 perceived as highly and 2 as less useful, were incorporated into an experimental government search system for comparison to a baseline system not providing facet capabilities. An experimental user study employing these systems found that there were some differences in the perceived usefulness and actual use of facets. Specifically, the audience facet, which received low usefulness scores in the questionnaire, was used quite frequently in the user study. Only few statistically significant differences between the baseline and experimental system were found. The most notable differences were found in Perceived Success, a measure of effectiveness, and Level of Satisfaction, a measure of satisfaction, between the first and third tasks performed in the experimental system, with the third task showing higher scores. iii Preface This dissertation is original, unpublished, independent work by the author, K. Kessler. Dr. Luanne Freund and Dr. Rick Kopak provided suggestions in terms of methodology and approaches to data analysis for both studies. iv Table of Contents Abstract ............................................................................................................................................ ii Preface ............................................................................................................................................. iii Table of Contents ............................................................................................................................ iv List of Tables ................................................................................................................................ viii Acknowledgements ........................................................................................................................ xii 1 Introduction ................................................................................................................................... 1 2 Literature and Systems Review .................................................................................................... 4 2.1 Literature Review .................................................................................................................. 4 2.1.1 Overview. ....................................................................................................................... 4 2.1.2 Search user interfaces in interactive information retrieval. ............................................ 4 2.1.3 Information architecture and faceted classification. ....................................................... 5 2.1.4 Faceted search................................................................................................................. 7 2.1.5 Work and search tasks. ................................................................................................. 10 2.1.6 Methods employed in previous research. ..................................................................... 12 2.1.7 Findings of previous research. ...................................................................................... 13 2.1.8 Open questions and conflicts in literature review. ....................................................... 15 2.1.9 Aspects not covered by literature. ................................................................................ 17 2.1.10 Implications for study 1. ............................................................................................. 19 2.1.11 Implications for study 2. ............................................................................................. 20 2.2 Systems Review .................................................................................................................. 21 2.2.1 Overview. ..................................................................................................................... 21 2.2.2 Facets and filter categories in the government domain. ............................................... 22 2.2.3 Facets and filter categories in the library and information domain. ............................. 28 2.2.4 Facets and filter categories in commercial domain ...................................................... 31 v 2.2.5 Summary and implications for study 1. ........................................................................ 34 3 Study 1: Task-based Perceived Usefulness of Facets ................................................................. 35 3.1 Research Design .................................................................................................................. 35 3.1.1 Introduction. ................................................................................................................. 35 3.1.2 Recruitment and participants. ....................................................................................... 35 3.1.3 Design of data collection instrument. ........................................................................... 39 3.1.4 Data analysis. ................................................................................................................ 40 3.2 Results ................................................................................................................................. 41 3.2.1 Level of perceived usefulness of facets by type or task. .............................................. 41 3.2.2 Analysis of variance. .................................................................................................... 43 3.2.3 Comments by participants. ........................................................................................... 43 3.3 Summary ............................................................................................................................. 44 4 Study 2: Task-based Use of Facets ............................................................................................. 45 4.1 Research Design .................................................................................................................. 45 4.1.1 Introduction. ................................................................................................................. 45 4.1.2 Recruitment and participants. ....................................................................................... 45 4.1.3 Experimental system and tasks. .................................................................................... 47 4.1.4 Data analysis. ................................................................................................................ 52 4.2 Results ................................................................................................................................. 53 4.2.1 Actual use and perceived usefulness of facets. ............................................................. 53 4.2.2 Satisfaction. .................................................................................................................. 55 4.2.3 Effectiveness. ................................................................................................................ 55 4.2.4 Efficiency...................................................................................................................... 56 4.2.5 Sequence effects. .......................................................................................................... 57 4.2.6 Comments by participants. ........................................................................................... 58 vi 4.3 Summary ............................................................................................................................. 59 5 Discussion ................................................................................................................................... 60 5.1 Overview ............................................................................................................................. 60 5.2 Perceived Usefulness and Use of Facets ............................................................................. 60 5.3 Systems with Facet Capabilities versus Systems without Facet Capabilities ..................... 61 5.4 Summary ............................................................................................................................. 63 6 Conclusion .................................................................................................................................. 66 6.1 Summary ............................................................................................................................. 66 6.2 Limitations .......................................................................................................................... 67 6.3 Future research .................................................................................................................... 67 6.4 Implications ......................................................................................................................... 68 References ...................................................................................................................................... 69 Appendix A ? Summaries of Assessment of Use of Facets and Filters ......................................... 83 Appendix B ? Study 1: Basic Participant Information .................................................................. 97 Appendix C ? Statistics Canada CANSIM Excerpts ..................................................................... 99 Appendix D ? Study 1: Questionnaire Example .......................................................................... 101 Appendix E ? Study 1: Descriptive Statistics, Normality, and Variance Details ........................ 109 Appendix F ? Study 2: Basic Participants Information ............................................................... 112 Appendix G ? Study 2: System and Activity Assignment ........................................................... 114 Appendix H ? Study 2: Scenarios ................................................................................................ 115 Appendix I ? Study 2: Protocol of Activities in Experimental User Study ................................. 116 Appendix J ? Study 2: Pre-Questionnaire .................................................................................... 117 Appendix K ? in Study 2: Search Instructions and Questionnaires ............................................. 118 Appendix L ? Study 2: Post-Questionnaire ................................................................................. 120 Appendix M ? Study 2: Normality and Variance Details ............................................................ 122 vii Appendix N ? Study 2: Sequence Variance Details .................................................................... 126 viii List of Tables Table 1: Systems Review - Assessment of Use of Facet and Filter Categories in the Government Domain (AU, CA, U.K., U.S.) ...................................................................................................... 23 Table 2: Systems Review - Assessment of Use of Facets and Filter Categories in the Library and Information Domain ...................................................................................................................... 28 Table 3: Systems Review - Assessment of Use of Facets and Filter Categories in the Commercial Domain .......................................................................................................................................... 31 Table 4: Systems Review - Occurrence of and Scores for Facets and Filters on Amazon.ca ...... 32 Table 5: Systems Review - Occurrence of and Scores for Facets and Filters on eBay.ca ............ 33 Table 6: Study 1 - Mean Perceived Usefulness of Facets Across Task Types ............................. 42 Table 7: Study 1 - Mean Perceived Usefulness of Facets for Doing Task Type .......................... 42 Table 8: Study 1 - Mean Perceived Usefulness of Facets for Known-item Task Type ................ 42 Table 9: Study 1 - Mean Perceived Usefulness of Facets for Learning Task Type ...................... 42 Table 10: Study 2 - Summary of Variables and Measures Tracked ............................................. 52 Table 11: Study 2 - Comparison of Mean Ranks of Perceived Usefulness and Actual Use of Facets ............................................................................................................................................ 54 Table 12: Study 2 - Use of Facets by Type of Task...................................................................... 55 Table 13: Study 2 - Satisfaction ? Comparison of Mean by Facet Availability and Measure ..... 55 Table 14: Study 2 ? Effectiveness: Comparison of Mean by Facet Availability and Measure .... 56 Table 15: Study 2 ? Efficiency: Comparison of Mean by Facet Availability and Measure ......... 57 Table A1: Systems Review - List of Departments and Search Systems Reviewed ? Government of Australia.................................................................................................................................... 83 Table A2: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Australia .................................................................................................... 84 Table A3: Systems Review - List of Departments and Search Systems Reviewed ? Government of Canada ...................................................................................................................................... 85 Table A4: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Canada ....................................................................................................... 86 Table A5: Systems Review - List of Departments and Search Systems Reviewed ? Government of the United Kingdom ................................................................................................................. 87 ix Table A6: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United Kingdom .................................................................................. 88 Table A7: Systems Review - List of Departments and Search Systems Reviewed ? Government of the United States of America .................................................................................................... 89 Table A8: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United States of America ..................................................................... 90 Table A9: Systems Review - List of Departments Reviewed on Amazon.ca .............................. 91 Table A10: Systems Review - Summary of Facets and Filter Categories Used on Amazon.ca ... 92 Table A11: Systems Review - List of Departments Reviewed on eBay.ca .................................. 93 Table A12: Systems Review - Summary of Facets and Filter Categories Used on eBay.ca ........ 94 Table A13: Systems Review - Assessment of Facets and Filter Categories Included in Questionnaire ................................................................................................................................ 95 Table A14: Systems Review - Assessment of Facets and Filter Categories not Included in Questionnaire ................................................................................................................................ 96 Table A15: Study 1 - Frequency of Age Ranges of Participants .................................................. 97 Table A16: Study 1- Gender Distribution of Participants ............................................................ 97 Table A17: Study 1 - Student Status of Participants ..................................................................... 97 Table A18: Study 1 - Employment Status of Participants ............................................................ 97 Table A19: Study 1 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) ....................................................................................................................................... 97 Table A20: Study 1 - Self-Reported Skill Level of Participants in Searching the Internet .......... 98 Table A21: Study 1 - Student Status of Participants in Age Group 22 to 31 ............................... 98 Table A22: Statistics Canada - CANSIM Summary of Age Groups and Genders - Year 2012 ... 99 Table A23: Statistics Canada - Population 15 Years and Over by Highest Certificate, Diploma or Degree (2006 Census) ................................................................................................................... 99 Table A24: Statistics Canada - CANSIM Summary of Student Status for Age Group 15 to 29 Years - Year 2012 ....................................................................................................................... 100 Table A25: Statistics Canada - CANSIM Summary of Employment Status for Age Group 15 Years and Above - Year 2012 ..................................................................................................... 100 Table A26: Study 1 - Descriptive Statistics for Assessment of Perceived Usefulness of Facets 109 Table A27: Study 1 - tests of Normality for Perceived Usefulness Score .................................. 110 x Table A28: Study 1 - Independent-Samples Kruskal-Wallis test Statistics for Perceived Usefulness with Grouping Variable Type of Task ..................................................................... 110 Table A29: Study 1 - Content Characteristic Date Available: Mann Whitney U test for Pairwise Comparison Between Types of Tasks......................................................................................... 110 Table A30: Study 1 - Content Characteristic Date Created or Published: Mann Whitney U test for Pairwise Comparison Between Types of Tasks .................................................................... 110 Table A31: Study 1 - Content Characteristic Geographical Area (about): Mann Whitney U test for Pairwise Comparison Between Types of Tasks .................................................................... 111 Table A32: Study 1 - Content Characteristic Timeframe: Mann Whitney U test for Pairwise Comparison Between Types of Tasks......................................................................................... 111 Table A33: Study 2 - Frequency of Age Ranges of Participants ................................................ 112 Table A34: Study 2 - Gender Distribution of Participants ......................................................... 112 Table A35: Study 2 - Student Status of Participants ................................................................... 112 Table A36: Study 2 - Employment Status of Participants .......................................................... 112 Table A37: Study 2 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) ..................................................................................................................................... 112 Table A38: Study 2 - Self-Reported Skill Level of Participants in Searching the Internet ........ 113 Table A39: Study 2 - Student Status of Participants in Age Group 22 to 31 ............................. 113 Table A40: Study 2 - System and Activity Assignment ............................................................. 114 Table A41: Study 2 - tests of Normality for Satisfaction Measures ........................................... 122 Table A42: Study 2 - tests of Normality for Effectiveness Measures ........................................ 122 Table A43: Study 2 - tests of Normality for Efficiency Measures ............................................. 123 Table A44: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Satisfaction Measures with Grouping Variable Facet Availability ................................................................ 124 Table A45: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Effectiveness Measures with Grouping Variable Facet Availability ................................................................ 124 Table A46: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Efficiency Measures with Grouping Variable Facet Availability ................................................................ 125 Table A47: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence ...................................................................................................... 126 xi Table A48: Study 2 - Mann Whitney U test for Pairwise Comparisons of All Measures Between Types of Tasks ............................................................................................................................ 126 Table A49: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Baseline System .......................................................... 127 Table A50: Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Baseline System ............................................................................ 128 Table A51: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Experimental System .................................................. 128 Table A52: Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Experimental System .................................................................... 129 xii Acknowledgements First and foremost, I would like to thank my dissertation supervisor Dr. Luanne Freund and my dissertation co-supervisor Dr. Rick Kopak for their comments, particularly in regards to their suggestion about methodology and data analysis. This research would not have been possible without the respondents taking time to complete the online questionnaire and participants in the user study. I would like to thank them for investing their time in contributing to this research. I would also like to thank the faculty and staff of the School of Archival, Library and Information Studies at UBC for their insights and support in all activities related to my study program. My gratitude goes to the two proof readers who spent their valuable time in mainly finding missing commas. Last but not least, I am sending a big thank you to the residents and staff (office and kitchen) of St. John?s College (SJC) for making my stay there very memorable and pleasant. This helped a lot in providing a balance to the work involved in completing my degree program and this research. 1 1 Introduction Already more than 40 years ago, Nobel laureate Herbert Simon (1971) noted that ?a wealth of information creates a poverty of attention and a need to allocate ? attention efficiently? (p. 40). With the continuous stream of information, an individual has to cope with an attention paucity regarding each piece of information (Simon, Egidi, & Marris, 2008). Hence it is necessary for important information to stand out to receive a higher level of attention. This leaves us with the challenge to ?represent, access and use information? in the information age (Tunkelang, 2009, p. vii). Consequently, there is a need to support decision-making processes in different task scenarios by providing intuitive modes of interaction for non-expert users in the process of information-seeking (Ben-Yitzhak et al., 2008). To handle the ?gargantuan volume? of online content and the significantly increasing number of user queries, search engines are currently considered to be the key tools for information seeking (Baeza-Yates & Ribeiro-Neto, 2011), but there are concerns that the commonly known and used models of search engines are not providing sufficient support for different types of tasks. Stoica, Hearst, and Richardson (2007) note that the base assumption is made that a standard search interface consists of a text query box and a result list. But is this assumption still valid? Only using a text query box and a result list seems to be a challenge, particularly for information about which individuals only know very little or nothing at all. This can be exemplified by Fountain?s (2001) finding that the average citizen is overwhelmed by the amounts of electronic data governments provide to them. Important qualities of information have to be exposed to capture more attention. Facets support this process by providing entry points to document characteristics (Li & Belkin, 2008). By doing so, facets not only allow for a different way of searching for content, but also result in a learning experience about content characteristics and an increased knowledge about the entire set of content beyond the few items actually reviewed more closely by the searcher. As outlined by Tunkelang (2009), the use of faceted search features is prevalent in the discovery of online content and has proven to provide a \"more effective information-seeking support to users than best-first search\" (p. vi). So far studies have focused on e-commerce and site search, but do not seem to have reviewed a broader scope focusing on different types of tasks. Hearst (2008) proposes that an extension of the faceted model to handle 2 complex content collections while not diminishing the proven usability of faceted search is needed. While innovations have been proposed, it is not yet known whether they improve usability and the number of facets that can be easily handled by the user. This research project is aimed at investigating the impacts of faceted search considering different types of search tasks and heterogeneous environments. Search systems are usually used in the context of completing a certain type of search task, for example a ?Known-Item? search task, which is aimed at finding a specific document. Types of search tasks vary in complexity and have shown to influence search behaviour. Faceted search in its basic form is understood as an initial free-text search which is supplemented by faceted navigation with elimination of facet values that would return an empty result set (Tunkelang, 2009). Usually, a specific set of facets is employed for searching a website or online store and this set remains static independently from what the user enters as initial free-text query. Hence, a specific set might not be applicable beyond the content niche a web site or online store is serving. But what about a dynamic adaptation of the facet set in a more heterogeneous information environment depending on the type of task performed? To address this issue in more detail the following research questions, hypotheses, and expected outcomes for the current study are listed below. Research questions to be answered: 1) Considering types of search tasks, is there a difference in the perceived usefulness of facets for discovery of online content? Does the actual use of facets in search systems vary by type of task? 2) Are static facets and dynamic task-dependent facets useful for discovery of online content? Hypotheses: 1) Search systems providing faceted search lead to a higher user satisfaction compared to search systems without this capability. 2) Search systems providing faceted search lead to a higher effectiveness compared to search systems without this capability. 3) Search systems providing faceted search lead to a higher efficiency compared to search systems without this capability. 3 Expected outcomes of research project: 1) Determination of facets perceived as most useful in search for online content, overall and by type of search task, as well as validation of perceptions through user experiments. 2) Determination and discussion of differences in user satisfaction, effectiveness and efficiency between systems without facet capabilities and systems with facet capabilities. This thesis is organised in the following way. Chapter 2 reviews the literature pertinent to search user interfaces, information architecture, faceted classification, faceted search, and work and search tasks. It also presents the results of a review of facet categories used in search interfaces across the library, government and e-commerce domains. Both literature and systems are scrutinized to inform the design of the research methodology. Chapter 3 covers study 1 of the research project, a questionnaire-based online survey, which focuses on answering research question 1, while chapter 4 outlines study 2, an experimental user study focusing on the government domain, which examines both research questions and rests the hypotheses of this research. Both chapters outline the research design, and present the data analysis and a summary. Chapter 5 outlines the answers to the research questions and discusses the validity of the hypotheses and salient findings in a broader context. Chapter 6 concludes the research by summarizing the limitations of this research, indications for future research, and implications. This work is part of a broader project studying information access and use in e-government (e-informing the public) conducted by the Digital Information Interaction Group (DiiG) at the iSchool of the University of British Columbia. 4 2 Literature and Systems Review 2.1 Literature Review 2.1.1 Overview. This sub-chapter aims at establishing a sufficiently broad but also focused review of the literature to inform the development of the methodology for studies 1 and 2. Having established the aim of the literature review, it is necessary to outline the topics that need to be covered to achieve these aims (Hart, 1998). Consequently this sub-chapter first addresses search user interfaces and interactive information retrieval to draw a connection between the noted paucity of attention and available remedies to address it. Based on reviewing these remedies, the underlying topic of information architecture and faceted classification are outlined in more detail. The topic of faceted search is then reviewed to exemplify the use of faceted classification in the digital world. Having established the connections between information retrieval, information-seeking and faceted search, work and search tasks are then explained in more detail. Then methods employed in, and findings of, previous research in interactive information retrieval are reviewed to facilitate summarizing the implications on studies 1 and 2. At the end of this sub-chapter, open questions and aspects not covered by the literature review are addressed. All individual reviews are then digested into concrete implications affecting studies 1 and 2. 2.1.2 Search user interfaces in interactive information retrieval. When discussing the discovery of online content, it is necessary to establish an understanding about the meaning of the terms information retrieval, information seeking, and user interfaces. Particularly the first two are often used interchangeably. As outlined by Vakkari (1999), a clear distinction between information retrieval and information seeking has to be made. The former mostly relates to the representation of, retrieval of and relevance assessments for documents, while the latter aims at gaining an understanding of how information retrieval activities can be supported in the context of information-seeking behavior. It can be suggested that information retrieval is a means to present relevant documents in the course of information seeking. Both areas of study can be combined into the research area of Information Search & Retrieval, which involves human subjects and not only algorithms (Toms, 2011). 5 Involving human subjects introduces the need to conduct investigations into the interactions with search user interfaces as a key element of the concept of interactive information retrieval (Saracevic, 2011). ?Interactive information retrieval can be represented as a set of relationships between human actors, information objects and systems? (Freund, 2008b, p.76). The development of systems supporting computerized information retrieval for humans and providing search user interfaces accessible to the wide public is a relatively young discipline which, in response to the exponentially growing amount of available online content, underwent a rapid development in the last two to three decades (Saracevic, 2011). Research into the individual areas of information retrieval and information-seeking behaviour already started in the first half of the 20th century with seminal works by Calvin Mooers (Cool & Belkin, 2011; Saracevic, 2011) and by Alfred Lotka and Samuel Bradford (Saracevic, 2011) respectively. Considering the rapid growth of content and the associated number of systems and their functions trying to support searching and finding relevant content, one fundamental research question has to be considered (Saracevic, 2011): What helps or hinders search? Hearst (2011) outlines that an answer to this question can only be given when considering the task context of user interactions. She states that in any case search systems should: ? Be human-centric rather than data-centric, ? Support users in refining their information needs, ? Provide facilities to formulate and reformulate user queries, visualize query results, select and assess retrieved content, understand content characteristics and record interactions. 2.1.3 Information architecture and faceted classification. It can be assumed that a big part in realizing this kind of system support is played by the inherent architecture of the content and its relationships. As outlined by Morville (2007, p.1), ?the combination of organization, labeling, and navigation schemes within an information system? constitutes an important consideration in terms of information architecture. Using classification as an intellectual instrument for understanding connections (Mills, 2004) can be seen as one measure to achieve an appropriate combination of these elements within an information system. Classification is seen as the recognition of groups of classes of objects which is fundamental for indexing and in turn necessary for search and retrieval. 6 As noted in the introduction, Li and Belkin (2008) state that facets support the process of exposing content characteristics to the user. For providing such functionality, it is necessary to establish a faceted classification system. Most scholars (e.g. Tunkelang, 2009; Broughton, 2006; Mills, 2004; Spiteri, 1998a, 1998b) attribute the development of the faceted classification system to Ranganathan (1950, 1951, 1963) and his works about the philosophy of the library and the development of the Colon Classification. An additional early contributor (Broughton, 2006; Spiteri, 1998b) is considered to be the Classification Research Group, which, according to Broughton (2006), stated the following reasons for contributing to the development of faceted classification: ?the display of useful generic relationships; full and accurate cross-referencing; accurate application of principles of division; a clear citation order; established rules for compounding; and an appropriate notation?(p. 50). Implementing a faceted classification system in a digital environment provides a more powerful base for search and retrieval. The use of faceted classification in a paper-based environment is challenging due to the linearity of arrangement, which means that an object is put only in one class (Mills, 2004). Hence, a high number of surrogates would be required to implement a faceted classification system (Broughton, 2006). In the digital environment linearity tends not to be a big concern, but the adequate description of objects and the provision of tools for accessing these objects, based on their description, become more important. To arrive at appropriate object descriptions the process of faceted classification, as summarized by Spiteri (1998b) and based on Ranganathan?s (1963) work, analyses the subject area in which the objects are situated to establish individual concepts and then synthesize similar ones into compound objects. Faceted classification is seen to be superior over relatively unstructured indexes and rigid taxonomies (Tunkelang, 2009). Taxonomies are ontologies in which only ?is-a? relationships are represented. An ontology represents a set of concepts and their relationships (Gruber, 1993), but hierarchies within an ontology result in only one path leading to a specific node; hence, it is difficult to integrate compound concepts. As noted by Spiteri (1998a), due to the Library of Congress Classification and the Dewey Decimal Classification being enumerative systems, it is very difficult to express compound objects using them. As each facet in a faceted classification system can be seen as forming a taxonomy, it is possible to overcome these limitations in expressing compound objects. By providing this solution, a faceted classification system 7 ?provide[s] an overview of results and incorporate[s] clickable categories into search results? (Kules, Capra, Banta, & Sierra, 2009, p. 313). As part of this system ?facets refer to categories used to characterize information items in a collection. A facet can be flat or hierarchical; in either case, a set of labels is associated with each facet? (Hearst, 2008, p. 1). Through these characteristics the assertion can be made that faceted classification is integral to the majority of approaches to information retrieval (Broughton, 2006). 2.1.4 Faceted search. Definition. Faceted search combines direct text search and navigational browsing through a faceted classification system (Tunkelang, 2006). To distinguish more clearly between the terms search and navigation, Hearst (2011) notes that searching is defined as typing a keyword query while navigation exposes the structure of the content to the user by making it traversable. Navigation is often also called browsing. Faceted search is used for ?integrating navigation and search? (Hearst, 2008, p.1). To be more precise, the navigation enabled through facets actually is a multifaceted navigation through multiple taxonomies (Ben-Yitzhak et al., 2008). Tunkelang (2009) summarizes the aim of faceted navigation as a means of accessing and using information represented by a faceted classification system. Tunkelang (2009) further elaborates that faceted search succeeds parametric search, which provides a visual interface for searching a faceted content collection via connecting constraints through Boolean logic. However, parametric search does not provide the possibility for keyword searches and does not offer guidance to the user about which queries are possible. While parametric search can be seen as a predecessor to faceted search, the term view-based search suggests an implementation of faceted search and hence can be considered a synonym. Background. For establishing a faceted classification system, the structured metadata within a content collection can be used (Kules et al., 2009). With this in mind, the resulting faceted structure can be seen as similar to a field-based structure in databases (Broughton, 2006). The model of the underlying metadata needs to be sufficiently simple for easy navigation but also sufficiently rich for flexible navigation (Hearst, 2008). Ideally, each item in a collection has multiple facet labels (Hearst, 2008) which is, as described in 2.1.3, contrary to a strictly hierarchical classification 8 system in which items are only assigned to one class, for example topic, date, and format. Hence, the assumption is made that records can be organized in multiple independent facets/taxonomies (Tunkelang, 2006). Their function as taxonomies distinguishes them from simple content filters that are usually employed in a static way independently from a collection?s actual content characteristics. The best results in designing faceted search systems can be achieved when facets represent separate concepts and their assignment to items can be used for combining and matching them (Hearst, 2008). Consequently, the two main pillars of faceted navigation are a multi-dimensional and multi-hierarchical scheme (Adkisson, 2005) of metadata and an appropriate document mapping (Ben-Yitzhak et al., 2008). Having established an appropriate faceted classification system, a search user interface is required to enable the use of the facets. It basically needs to provide the means for a user to select (or exclude) facet labels. Selecting a particular facet label results in limiting the result set to all objects within a content collection that match this label (Hearst, 2008). Usually selecting another facet label will create a conjunction (logical AND) between these two labels and result in only displaying the items that match both labels. Tunkelang (2009) states that the use of inclusive disjunction (logical OR) can be considered as well. The use of dynamic queries by employing interactive and visual control of parameters (Shneiderman, 1994) and the elimination of zero-hit queries to generate query previews (Donn, Plaisant, & Shneiderman, 1996) are considered to be the most important aspects in facilitating faceted navigation. Importance of faceted search within search systems. Based on previous research, the assumption can be made that faceted search provides several important advantages over simple keyword search. Kules et al. (2009) summarize these advantages based on eye gaze tracking experiments: ? Facilitation of exploratory search, ? Support of complex information seeking activities, ? Functioning as an alternative to query reformulation by allowing for mix of searching and browsing, ? Informing searcher?s knowledge about domain and improving their understanding of their information needs. 9 Faceted search allows for progressive query refinement by presenting a complex information space in way that enables intuitive exploration of the content collection (Ben-Yitzhak et al., 2008; Tunkelang, 2009). It allows for navigation across multiple facet hierarchies via drill-down refinement and roll-up generalization (Adkisson, 2005) by giving users an indication of the effect their actions will have (Ben-Yitzhak et al., 2008). Its use has proven to facilitate exploration and discovery (Hearst et al., 2002; Yee, Swearingen, Li, & Hearst, 2003; Kules and Shneiderman, 2008). This is particularly applicable to larger content collections (Tunkelang, 2009). Examples of successful use of faceted search can be found within the indexing practices of library and information organizations, as well as the product information provided by commercial web presences (Broughton, 2006). Innovations in faceted search. As the field of interactive information retrieval is quickly developing based on the rapid content growth, so are the front-end and back-end solutions for implementing faceted search systems. Hearst (2008) noted that two major front-end innovations are being pursued. Firstly, implementing a separate keyword search for all facets and incorporating an auto-suggest feature so that users can search for facet labels. In an earlier publication, Tunkelang (2006) noted that this innovation will suffer from the vocabulary problem as users might have different concepts of labels in mind than the information architect designing the facet hierarchy. Hearst (2008) elaborates that only very few if any studies have been performed to investigate the usability of this auto-suggest feature for facet labels. The second innovation is the automatic selection and reordering of facet labels based on keyword searches. Hearst also suggested that the elimination of facets based on the keyword search term is an innovative feature. Beyond the suggestions by Hearst, Ben-Yitzhak et al. (2008) proposed co-relating facets and displaying additional information beyond the number of results matching a facet label, such as average price if it is a purchasable object or average number of pages for documents. They concede that one major challenge is the mapping of documents to facet labels as this is usually perceived as a manual process performed by the creators of the object. Broughton (2006) suggests that automatic mapping based on algorithms could be a solution. For example, Dakka, Ipeirotis, and Wood (2005) and Dakka, Dayal, and Ipeirotis (2006), as well as Stoica et al. (2007) proposed algorithms to extract facet hierarchies based on text mining and natural 10 language processing to remedy the challenges related to the availability of metadata. However, it seems that the dynamic adaptation of the facet set in a more heterogeneous information environment depending on the type of task performed does not seem to be a major suggestion for innovation in this field. This is surprising given the importance of tasks as a motivation in Interactive information retrieval research. 2.1.5 Work and search tasks. Considering the connection drawn between information retrieval, information-seeking and faceted search, a more detailed review of tasks in this context is warranted. Morville (2007) notes that ?the structural design of an information space is to facilitate task completion and intuitive access to content? (p. 1). This statement is not made lightly as other researchers have determined that the effects of tasks are potentially important when designing information systems (J?rvelin & Ingwersen, 2004). This relates to an asserted relationship between types of tasks and the behaviour of information seekers (Vakkari, 2003), such as the varying length and number of queries in information-gathering (Toms et al., 2008). Ultimately, the complexity and structure of a task lead to the suggestion that the type of task is a consideration in the assessment of documents by information seekers (Vakkari, 1999), e.g. the type of document, its genre, is more important when performing ?Doing tasks? (Freund, 2011). Hence, it can be concluded that the nature of the task influences information seeking behaviour (Li & Belkin, 2008). The term task itself is not as clear-cut as it initially seems and hence needs to be dissected for an appropriate understanding of the outlined effects. According to Toms (2011), tasks can be considered as the process of performing activities to move from a goal to an outcome. These activities are usually performed in a specific domain or environment that constrains their completion (Taylor, 1991; Vicente, 1999; Bystr?m & Hansen, 2005). Thus, it can be assumed that tasks consist of multiple levels of sub-tasks, have varying degrees of complexity, and are embedded in distinctive ?problem structures? while being influenced by the actor?s prior knowledge, experience, and cognitive capacity (Vakkkari, 1999). Particularly, a lack of knowledge usually necessitates an information seeking process as a sub-task. Conclusively, tasks, more specifically work tasks (Hansen, 1999), can be subdivided into different activities to be completed by individuals during the course of work or life (Li & Belkin, 2008). Work tasks can trigger information needs and in turn trigger information seeking tasks as a sub-division 11 (Toms, 2011), such as citizens? information seeking in dealing with government agencies to solve everyday life information needs (Savolainen, 1995). Hence, it is necessary to consider the work and information seeking tasks when trying to determine the information need (Ingwersen & J?rvelin 2005). This has a more profound effect in an information-intensive environment in which work tasks essentially become information work tasks that need to be fulfilled in professional (Li, 2008) and in personal life (Wildemuth & Freund, 2012). Having determined that a distinction has to be made between the overarching work task for accomplishing something and a potentially required information seeking task and considering the unique ?problem structure? of different work tasks (Vakkari, 1999), the need to define types of tasks encompassing the same or a very similar ?problem structure? arises. For most if not all information seeking tasks, interactions with information retrieval systems are required (Toms, 2011). As summarized by Bystr?m and Hansen (2002, 2005), Li and Belkin (2008) and Wildemuth and Freund (2012), there are different kinds of search tasks that can be grouped according to their characteristics. As outlined in earlier works by Marchionini (1989) and Walker, Janes, and Tenopir (1999), one of the most profound distinctions is the level of specificity of a search task. Specific search tasks can be considered as closed-ended (Marchionini, 1989) and include search tasks such as known-item search, factual or fact-finding search, navigational search, and simple question-answering search. In contrast, general search tasks can be seen as open-ended (Marchionini, 1989). Deciding, doing, learning and problem-solving as defined by Freund and Berzowska (2010, p.3) constitute general search tasks and are essentially more complex and exploratory in nature. Considering the purpose of exploratory search as part of information seeking in complex environments and when searchers? expertise is limited, a definitive connection to faceted search can be drawn. Exploratory search tasks are employed in the information seeking process when the lack of domain knowledge (Vakkari, 1999) cannot be remedied by simply finding an answer, but by getting a broader sense of the problem and its surrounding structure to address the complexity of a task (Bystr?m & J?rvelin, 1995). White, Kules, Drucker, and schraefel (2006) define exploratory search as when the searcher ?lack[s] the knowledge or contextual awareness to formulate queries or navigate complex information spaces, the search task requires browsing and exploration, or system indexing of available information is inadequate? (p. 37). As such, it 12 involves learning and investigation (Marchionini, 2006) employed for development of intellectual capabilities (White & Roth, 2009). Types of specific search tasks, such as known-item, navigational and question answering search can be part of an exploratory search process (Kules et al., 2009). As summarized by Wildemuth and Freund (2012) exploratory search tasks can usually be characterized as open-ended, uncertain, ill-structured, and aiming at retrieving multiple items. They also outline the usual characteristics of the exploratory search process as dynamic, multi-faceted, complex, and accompanied by additional cognitive behaviours. It becomes clear that researching exploratory search tasks is an equally challenging as well as important behavioural process to investigate due to its complexity and uncontrollability. The importance of developing methods and designing experimental search tasks to induce exploratory search behaviour while maintaining a sufficiently high degree of control need to be considered when conducting studies in interactive information retrieval (Kules & Kapra, 2012; Wildemuth and Freund 2012). 2.1.6 Methods employed in previous research. Research in interactive information retrieval has made wide use of simulated work tasks as introduced by Borlund (2000). Participants are usually presented with a set of search tasks (Kules & Shneiderman, 2008; Yee at al. 2003). Tasks are usually described as requirements and aims to accomplish a ceratin goal. The process of achieving these aims can be seen as a surrogate for the real-world behaviour of a participant (Bystr?m & Hansen, 2005). The challenge in using work tasks lays in their realistic and representative design (Kules et al., 2009). A realistic and representative design is usually approached by embedding tasks into scenarios. For example, the five information work tasks, fact-finding, deciding, doing, learning, problem-solving, identified by Freund (2008a), were each presented in four different scenarios (Freund, 2008b). Kim (2012) identified and reviewed 129 experimental and non-experimental studies using scenarios. Her findings outlined that many studies employ interactive information retrieval using scenarios to evaluate the effectiveness of systems in terms of search performance and assessment of relevance. Often slightly different instances of a prototype system were developed and compared with each other. As further outlined by Kim (2012), the advantages of using scenarios in this process lay in the indirect elicitation of responses, a higher efficiency compared to observational studies, and higher internal validity due to creating standardized conditions 13 particularly in experimental research designs. On the other hand, the external validity of a study might suffer as scenarios are artificial, similarly to what Kules et al. (2009) noted about tasks; hence individuals might behave differently when faced with a similar situation in real life. Beyond the challenge of designing artificial, but still realistic and representative tasks and scenarios, other effects, such as the type of task and the genre of the assessed documents, need to be considered as well. For example, in an experimental study with 25 participants conducted by Freund and Berzowska (2010), five task-based scenarios were presented to each participant to determine the impact of the tasks on the assessment of the document?s usefulness. Scenarios were rotated to remedy any order effects and participants rated the scenarios as quite realistic. The criteria for assessing the usefulness of documents were found to be inconsistent across tasks. In this context, it could also be observed that the genre of documents can impact usefulness assessments (Freund, 2011). In addition to employing task-based scenario approaches in user studies, log analysis is being used to derive supplemental measurements while the employment of eye gaze tracking has been a more recent development. For example, user studies such as by Kules and Shneiderman (2008), Capra, Marchionini, Oh, Stutzman, and Zhang (2007), and Yee et al. (2003) employed small-scale analysis of logs in addition to measuring task completion and user satisfaction. More recently, studies, for example by Kules et al. (2009) employed eye gaze tracking in combination with stimulated recall interviews and direct observation by measuring the seconds of eye gaze on 8 areas of an experimental search system and comparing it to the participants? perception as indicated in questionnaires. 2.1.7 Findings of previous research. While simulated work task scenarios are widely used in studies in the field of interactive information retrieval, they vary widely in their focus and findings. As outlined earlier Freund and Berzowska (2010) focused on what criteria are being used for assessing the usefulness of documents depending on the type of task. The main findings of this study suggest that people look for information that is specifically matching their needs from an individual perspective and that, amongst other factors, different interpretations of the scenarios influenced the document assessment. 14 Often work task scenarios are embedded in studies comparing aspects of different systems. Content formats, domain, generation of information structures, and their relationships are examples for focus areas of previous research. For example, Yee et al. (2003) conducted a comparative within-subjects usability study of an experimental user interface based on conceptual dimensions (facets) and a standard image search interface for exploration of a collection of 35,000 art history images. 32 art history students performed 4 tasks, 2 unstructured and 2 structured, using both search interfaces. The study indicates that although the experimental interface responded slower it was preferred and that category-based approaches using faceted systems can provide a successful way for accessing images and a higher user satisfaction. English, Hearst, Sinha, Swearingen, and Yee (2002) also conducted research in the area of image retrieval. Their study focused on supporting non-professional searchers in the context of rich information seeking. For this purpose, two usability studies were performed of which the first, having 11 participants, did not result in a clear system preference. For the second study, the elements considered most useful by participants in study 1 were used to create two new interfaces. 19 participants were asked to complete three image search tasks using both systems. English et al. (2002) established the main conclusion that facets are useful for creating structures for navigation. This finding bears similarity to the suggestion by Kules et al. (2009) that facets are important in the context of exploratory research. As summarized in 2.1.6 Kules et al. (2009) conducted a study comparing faceted libraries catalogs using eye-gaze tracking, recall interviews, and observations. While Uddin and Janecek (2007) concluded in a 19 participants within-subject study that faceted systems are improving efficient access to information, search success, search flexibility, learning, relevance of search results, and user satisfaction when compared to single classification systems, the challenge of generating facet labels and structures remains. Pratt, Hearst, and Fagan (1999) compared three different approaches to addressing this challenge: dynamic categorization of results, clustering of results, and ranking of results. They used knowledge about user queries and domain terminology to dynamically categorize results into a hierarchical faceted organization. 15 users were asked to use three systems, each based on one of the mentioned approaches, to determine the support each system provided for learning, whether questions were answered efficiently and easily, and whether participants perceived the search experience as 15 satisfactory. The Main finding of Pratt et al. (1999) was that more results were viewed with the system using dynamic facet categorizations. Some of the different, and sometimes contradictory, results in investigating whether users prefer facets or not might be explainable through a finding by Capra et al. (2007). Having conducted a two part study investigating relationships between search tasks, information structures, and interface design, they concluded that preference might be given to familiar systems providing traditional interfaces including keyword search and result lists. For this study 28 participants performed three search tasks in a between-subject study across three systems, while 12 participants different from the 28 participants conducted three kinds of search tasks in all three systems. 2.1.8 Open questions and conflicts in literature review. Faceted search. While faceted search can provide advantages over simple text queries, there are challenges associated with the implementation of search systems employing facets. As noted by Capra et al. (2007) facets are not always the preferred search tool for users although they offer a higher degree of effectiveness. One major cause for this might be related to the vocabulary problem demonstrated by Furnas, Landauer, Gomez, and Dumais (1987). Tunkelang (2009) elaborates on this by pointing out that while the user is being guided through the information space by faceted navigation, the underlying metadata is usually created based on the conception of its creators and might mismatch the terms searchers are actually looking for. An example for this mismatch seem to be the terms used by the government to communicate to members of the public which in turn can result in a mismatch of facet label assignments for e-government documents (Freund, 2011; Freund, Berzowska, & Hopton, 2011). So it seems that the concepts associated with facets and facet labels are open to interpretation depending on specific situations and perceptions of individuals (Broughton, 2006). Ultimately this leads to the challenge of creating sufficiently structured metadata for creating facets, hierarchies, and labels therein, as well as assigning appropriate labels to items in a collection (Stoica et al., 2007). Considering the vocabulary problem it becomes obvious that the design of the facets and the search system supporting them is of great importance. It is necessary to understand the effects of 16 facets on search activities and the design of search systems (Kules et al., 2009), but there seems to be a trade-off between the usability of the interface of a faceted search system and the support for larger and heterogeneous facet hierarchies (Hearst, 2008). Hearst (2008) elaborates on the concern of diminished usability by outlining that while faceted navigation improves flexibility within a single content collection, its usability and usefulness in large-scale projects might be reduced for non-expert users by the overwhelming number of facets and labels. Tunkelang (2009, p.51) on the other hand points out that ?we need to face the possibility of a large number of heterogeneous, interdependent facets? in faceted search systems due the ever-growing amount of content available to searchers. Hence, the choices in designing faceted search systems need to be different depending on the area of application and the needs of the users (Tunkelang, 2009). But aligning faceted search systems depending on the area of application might lead to only very specific real-world scenarios and in turn to a set of non-transferable facet hierarchies. As Hearst (2008) notes, in real-world scenarios there are many cases when some facet concepts can only be combined with a small sub-set of other facet concept, hence innovative design solutions have to be developed to address this challenge. Ben-Yitzhak et al. (2008) goes beyond this and suggests that ?another shortcoming of faceted search is that its basic data model, where documents are associated with sets of values across several independent facet hierarchies, is too restrictive to model some real-life data? (p. 33). He and his colleagues claim that the faceted ?standard model? implies that any object in a collection can be available in all combinations of facets so that any value in one facet can exist with any value in another facet. But they note that such independence does not necessarily exist by giving an example that facets made available for a product could be colors red and blue and sizes small, medium and large, but in reality the product is only available in color red and size small, and in color blue and size large. But, while there are obviously limitations in which facet labels can be used in conjunction with each other, this problem is at least partially solved by not displaying empty facets and empty facet labels. Reviewing the open questions and conflicting views on faceted search systems, it can be confirmed that there is a wide-spread acknowledgement of the advantages and possibilities provided for searchers, but that there are still big challenges to be solved to unleash the full potential of facets. Tunkelang (2009) summarizes the challenges and debates regarding faceted search that need to be addressed as scalability of storage, efficiency of query processing, 17 availability of metadata, information overload by a too high number of facets and facet labels, the vocabulary problem, different types of entities, and effective organization and presentation of facets and facet labels. Tasks and methodology. Similarly to faceted search having advantages, the use of tasks as part of the methodology in user studies has proven to show potential for eliciting particular kinds of search behaviour and in turn suggestions for system improvements. But at the same time the concept of tasks and associated scenarios introduces challenges. According to Toms (2011) the main problem is the broad and inconsistent range of categorizing work tasks. She exemplifies this by asking what the difference between decision making and problem solving is, and suggests that consideration must be given to their contrasting meanings in different domains. Ultimately she notes the need for a formal model for both work tasks and search tasks. Supplementing such a formal task model would be the establishment of a set of scenarios and tasks highly relevant in real-life based on observation of people and a large-scale analysis of transaction logs (Bystr?m & Hansen, 2005). Going beyond this, it has also been suggested that no formal validation of the use of work tasks in interactive information retrieval has been performed (Borlund & Schneider, 2010). This problem might be aggravated by the effect of researcher-generated and imposed tasks on participants? perceptions (Li & Belkin, 2008) and by participants of studies mostly being experienced users (Kules et al., 2009). Although it has been stated that experimental user studies employing task-based scenarios should not be longer than an hour (Kim, 2012), the question seems to remain when studies become too long and adversely affect the motivation for study participants to invest a significant amount of time in the process of completing a search scenario (Wildemuth & Freund, 2012). 2.1.9 Aspects not covered by literature. Genre theory and faceted classification. There seems to be a connection between genre theory and faceted classification. As outlined by Orlikowski and Yates (1994), genres are identified by means of form, content and purpose. This seems to be very similar to the process of faceted classification. Genre can be seen as the common ground in interaction and communication processes (Freund & Nilsen, 2008) in which 18 information objects take on a specific purpose in the context of social acts (Freund, Berzowska, & Hopton, 2011), while classification can be considered as recognizing classes objects belong to (Mills, 2004). As such, both can be suggested as very important for indexing content and making it accessible for search and retrieval. So it seems that genre theory focusing on the communicative aspects of a document could be seen as part of faceted classification. While genre seems to take into account several document characteristics, hence facets, it can in turn be represented as a facet and instances of genre as facet labels. As outlined by Freund (2011), genre can impact the assessment of usefulness of documents. Considering this effect, it can be assumed that concepts in a faceted classification system can also impact the assessment of usefulness. Alternative subject recruitment. A practical challenge faced in most studies involving human participants is subject recruitment. It can be suggested that most if not all studies in interactive information retrieval use more or less traditional ways of recruitment, such as posters in public places and sending invitations via mailing list. These recruitment approaches are usually limited in reach and often lead to students being participants in studies which in turn leads to issues in the generalizability of results (Druckman & Kam, 2011; Morton & Williams, 2010). Liu, Lease, Kuiper, and Bias (2012) conducted a usability study of a school web site using a convenience sample of students and a crowdsourcing recruitment approach via MTurk and CrowdFlower. Comparing both samples, they suggest that crowdsourcing could be an alternate approach to the usually more expensive and time consuming sampling via traditional methods. But they also point out that only a very limited number of potential participants are registered with the used crowdsourcing platforms, and that many participants perform their work poorly and most of them are from the United States. Precisely because of these limitations Samuels and Zucco (2012) considered Facebook for their crowdsourcing recruitment approach. Having a much higher number of reachable users and a more fine-grained possibility for targeting particular demographic groups they decided to use Facebook for a study in the area of political science with more than 3,000 participants from Brazil with a much lower cost than traditional ways of recruitment have incurred. As their field of study is not related to interactive information retrieval at all it remains to be seen whether a successful adaptation of the recruitment approach is feasible. 19 2.1.10 Implications for study 1. Considering the body of literature reviewed, this research being part of a wider research project, and this being a multi-part study, the following implications on study 1 of this thesis, a questionnaire-based online survey to validate the perceived usefulness of filter/facet categories, can be derived: ? Recruitment approaches: o At least one traditional and one crowdsourcing recruitment approach need to be used to reach a higher number of participants and remedy the effects of convenience sampling. o Poster recruitment has been selected as it has been successfully used in previous studies being part of the e-informing the public. o Facebook ads, as demonstrated by Samuels and Zucco (2012), have been selected as crowdsourcing recruitment approach. ? Selection and design of tasks: o While the study only presents abbreviated examples of task-based scenarios to participants, the tasks should still be relevant and clear to study participants (Borlund, 2003). o Three of the five types of search tasks, Doing, Known-Item, and Learning, identified by Freund (2008a) are selected to reduce the extent of the questionnaire and the time required to complete it, while still presenting task types with different degrees of complexity. o Tasks need to be embedded in scenarios (Freund & Nilsen, 2008). o As noted by Kim (2012), the order of presentation of scenarios should be randomized. ? Selection and design of scenarios: o Scenarios need to be designed to present realistic situations (Freund & Berzowska, 2010). o Different scenario domains should be used to not bias participants? responses to a particular domain. o Scenarios used by Freund and Berzowska (2010) for e-government context can be used as base for examples in this domain as they have been found to being quite realistic. ? Selection of facets: o As genre plays a role when assessing document usefulness (Freund, 2011), it should be considered for inclusion as facet into the study. o Facets with different levels of anticipated usefulness should be included in the questionnaire. 20 2.1.11 Implications for study 2. Considering the body of literature reviewed, this research being part of a wider research project, and this being a multi-part study, the following implications on study 2 of this thesis, an experimental user study presenting task-based scenarios and different instances of a prototype of a search system to participants, can be derived: ? Recruitment approaches: o As this study is lab-based, only local participants can be considered. Poster recruitment has been selected as it has been successfully used in previous studies being part of the e-informing the public. Other approaches allowing for recruitment of local participants will be considered as alternatives. o Participants will also be recruited from participants of study 1, if they indicated their availability for participating in study 2. ? Selection and design of tasks: o The same task types as in study 1, Doing, Known-Item, and Learning, are to be used. o Tasks need to be embedded in scenarios (Freund & Nilsen, 2008). ? Selection and design of scenarios: o Scenarios need to be designed in a way to present realistic situations (cp. Freund & Berzowska, 2010) and should be aligned to scenarios previously used in studies conducted for the e-informing the public project (where applicable). o As outlined by Kim (2012), participants in a study using scenarios must find the scenarios to be highly realistic so that they can identify with the situation. The description of the scenario needs to outline topic and context. A scenario-based experimental study should not take longer than an hour and the order in which scenarios are being presented should be randomized. o Scenarios used by Freund and Berzowska (2010) for e-government context can be used as they have been found to be quite realistic. ? Selection of facets: o Facets will be selected based on the outcome of study 1. o As this study is about assessing different system features, the facets with the highest usefulness are to be if applicable and technically possible. Other facets will be considered if they were expected to be of higher usefulness. 21 2.2 Systems Review 2.2.1 Overview. As outlined under 2.1.10 it is necessary to look at different domains for determining scenarios and to obtain indications for which facets might be of particular interest. Hence, it is necessary to identify appropriate domains and facets or filter categories employed within them. Filter categories are included in addition to facets as they can provide indications of what content characteristics are important in a particular domain. As outlined in 2.1.4, the major difference in facets and filter categories is based in facets functioning as taxonomies while simple content filters are usually employed in a static way independently from a collection?s actual content characteristics. Filters are usually displayed in an advanced search interface and require a direct text input by the searcher or selection from a drop-down menu, so they can be used before submitting a search. In contrast, facets are presented as navigable hierarchies that are usually presented after a text query search only displaying the relevant parts of the hierarchies. Faceted classification is often employed as part of the indexing practices within library and information organizations (Broughton, 2006). Broughton (2006) also notes that commercial web presences often use facets to display product information. Consequently, the library and commercial domains should be further investigated. Additionally content of government web sites can be suggested as a domain of interest. Fountain (2001) notes that the amount of electronic content provided by governments usually overwhelms the typical citizen. This happens although governments have or at least should have the priority to ensure citizens have an appropriate level of access to government services via the Web (Freund & Berzowska, 2010). It becomes obvious that there is a severe vocabulary problem between the terms and concepts used by government units and the needs and perceptions of the public (Freund et al., 2011). Hence, governments should be aiming to design systems able to provide citizens with the necessary tools to access this information. The following sections derive indications for importance of facets by reviewing four web-based systems in each of the identified domains. To derive indications of the importance of facets in the different domains, using a mere occurrence of a filter category or facet would not provide sufficient detail. Hence, a distinction needs to be made between different levels of use in the actual search systems, such as planned use according to system specifications or standards and providing the filter or facet to a user of 22 the system. While system specifications and metadata standards are mostly publicly available in the government domain, they are usually not available for commercial systems. While it seems that individual system descriptions or standards used are not listed for libraries the assumption can be made that they all use a very similar standard for electronically storing records. Consequently potential differences in determining the levels of use of facets or filter categories in different domains need to be accounted for. Hence, as outlined below the score of 1 can be assigned to a facet or filter category either based on system or content documentation, or occurrence in a search system. Facets and filter categories will be counted as an occurrence if they are explicitly visible in a search system or if a certain sub set of facet or filter labels can be found. The occurrence of facets and filter categories is scored in this way: ? A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. ? A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. ? A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. Based upon the review of four systems in each domain implications on study 1 are derived at the end of this sub chapter. 2.2.2 Facets and filter categories in the government domain. This section covers the search user interfaces of four governments: Australia, Canada, The United Kingdom, and The United States of America, presumably providing a big proportion of the national government content available in the English language. For each government the documentation describing the planned use of content characteristics and the actual availability of content characteristics to searchers are reviewed. Table 1 provides a concise overview of which facets or filter categories can be considered more or less important, while the following sub sections outline the major findings. 23 Table 1 Systems Review - Assessment of Use of Facet and Filter Categories in the Government Domain (AU, CA, U.K., U.S.) Facet/Filter Category AU CA U.K. U.S. Total Also referred to as Type - Document 2 2 2 3 9 Document Type, Resource Type, Genre Item - Format 3 2 1 2 8 File Type Subject 2 2 2 2 8 Person (about), Topic Language 1 3 1 2 7 Organization 3 1 3 7 Agency, Branch, Department, Sub-Department, Type - Category Time Frame 1 3 2 1 7 Coverage - Temporal Audience 2 1 2 1 6 Coverage - Spatial 1 2 1 2 6 Location (about) Date - modified 1 3 1 5 Creator 1 1 1 1 4 Author, Owner, Speaker Availability 1 1 1 3 Location (of object) Contributor 1 1 1 3 Date - published 1 1 1 3 Date - created Source 1 1 1 3 Date - extracted 1 1 2 Function 1 1 2 Activity Item - Extent 1 1 2 File ? Extent, Size Mandate 1 1 2 Publisher 1 1 2 Relation 1 1 2 Rights - Access Rights 1 1 2 License Status 1 1 2 Type - Aggregation Level 1 1 2 Collection Accessibility 1 1 Addressee 1 1 Aggregation 1 1 Coverage - Jurisdiction 1 1 Date - Availability 1 1 Date - Copyright 1 1 Date - issued 1 1 Date - licensed 1 1 Date - reviewed 1 1 Review Date Date - Validity 1 1 Digital signature 1 1 Disposal 1 1 Preservation 1 1 Rights - Rights Holder 1 1 Type - Service 1 1 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. The maximum score for a facet or filter category across all government organizations reviewed is 12. Abbreviations for government web presences: AU = Australia: CA = Canada; U.K. = United Kingdom; U.S. = United States Number of departments: n(AU) = 18, n(CA) = 21, n(U.K.) = 21, n(U.S.) = 17; Status as of April 22nd, 2013 Table 1: Systems Review - Assessment of Use of Facet and Filter Categories in the Government Domain (AU, CA, U.K., U.S.) Use of facets and filters in the web presence of the Australian Government. The National Archives of Australia (2010) published the Australian Government Locator Service Metadata Standard, to be used based on the needs of the government organisation, which specifies a comprehensive list of metadata elements based on extended Dublin Core. These include the metadata elements audience, availability, contributor, coverage (jurisdiction, spatial, 24 temporal), creator, date (Availability Date, Copyright Date, Creation Date, Issue Date, License Date, Modification Date, Validity Date), description, format (extent, medium), function, identifier for bibliographic citation, language, mandate (act, case, regulation), publisher, relation to other content, rights (Access Right, Rights Holder, License), Source, Subject, and Type (Aggregation Level, Category, Document Type, Service Type). These elements, with the exception of identifier which usually relates to one particular content object only, can be scored with a 1 as they indicate potential facets. While a search system across all government departments is provided1, the filter possibilities presented to the searcher are very limited. The site offers a high level organizational distinction between federal and state web sites, two document types (Publications, Media Releases), location by post code and six file formats. Only file format seems to be sufficient for being considered to receive a score of 3, while the others are too rudimentary to being considered for higher score than 2. Hence it is necessary to review the 18 main federal government departments, a complete list of which can be found in Table A1 in the appendix. Amongst these 18 departments the use of facets is very limited while the use of filters is somewhat more prevalent. The only departments presenting facets to searchers are the Department of Education, Employment and Workplace Relations2, the Department of Infrastructure and Transport3, and the Parliament of Australia search system4. The first uses audience, the second sub department (organization) and the third document type and date modified as facets. Only considering these facets is not sufficient to determine a scoring higher than 1 though. Reviewing all departments for filters categories presented to searchers, seven departments do not present any, while most others present one or more of 7 filter categories ? the number in brackets denotes the total number of departments employing a filter: audience (2), document type (3), file format (1), keywords (1), organization ? sub department (6), subject (5), and time frame (1). Table A2 in the appendix outlines the form of use and the resulting scoring for each facet/filter category in more detail. 1 http://australia.gov.au/funnelback/search?collection=gov_all&coverage=all&form=simple&gscope1=&query=test&query_prox=&query_and=&query_not=&query_phrase=&sort=&num_ranks=&meta_f_sand=&advancedSearch= 2 http://deewr.gov.au/search/site 3 http://search.infrastructure.gov.au/search/search.cgi?collection=Infrastructure&form=advanced 4 http://parlinfo.aph.gov.au/parlInfo/search/search.w3p;adv=yes 25 Use of facets and filters in the web presence of the Canadian Government. The Treasury Board of Canada Secretariat (2012) established a metadata scheme based on Dublin Core to be applicable to all Government Canada online content. Focus is put on the use of the elements audience, file format, location (coverage ? spatial), subject, and document type. Hence, these elements can be scored with at least a 1. Search systems provided by individual government agencies and departments partially provide different kinds of facets or filter categories to searchers. Only two of the reviewed 21 departments, a complete list of which can be found in Table A3 in the appendix, present facets to the searcher. Agriculture and Agri-Food Canada5 displays a subject facet after a search query has been entered. The Department of National Defence6 employs a comparably comprehensive faceted search system covering the facets of document size (file extent), document type, file format, function (activities), language, location (coverage spatial), source and subject. Out of the 21 departments reviewed five do not provide any obvious facet or filter functionality while many of the other departments are favoring date modified, document type and time frame within a set of 13 employed filter categories. The following filter categories are being used in the 16 departments that employ filters - the number in brackets denotes the total number of departments employing a filter: author (1), contributor (1), date created (1), date extracted (1), date modified (9), date reviewed (1), document type (6), file format (2), keywords (2), language (4), location ? coverage spatial (1), subject (2), and time frame (9). The filter categories date modified and time frame are assigned a score of 3 as they are being used in a significant plurality compared to all other filter categories. Although language is only presented as filter in 4 cases, it is still being assigned a score of 3 as almost each and every web page in the domain of Government Canada provides the possibility to switch between the English and French versions. Table A4 in the appendix outlines the form of use and the resulting scoring for each facet/filter category in more detail. 5 http://srch-rech.agr.gc.ca/srch-rech/aafc-aac/search-recherche.jsp?advanced=true&FileFormatBox=html&lang=eng 6 http://www.index.forces.gc.ca/Srch.aspx?lang=en-CA&Scrn=Adv 26 Use of facets and filters in the web presence of the U.K. Government. An e-Government Metadata Standard based on Dublin Core has been published by the Cabinet Office (2006, p. 10) and defines the elements creator, date (issued), subject and title as mandatory metadata. Title cannot be considered as facet or filter as it is usually not following a taxonomy that can be transferred into a faceted classification system. The elements accessibility, identifier, and publisher are indicated as mandatory if applicable to the particular case. Identifier cannot be considered as a facet as it usually only returns on specific item. Coverage (spatial) and language are recommended while additional 16 elements are specified for optional use of which the element description cannot be considered as facet or filter category due to the same reasons as for title. The main page of the government web presence7 provides the possibility to browse, so essentially filter, by topic (subject), while the search system of the GOV.UK single government website8 does not provide any facets or filters at all and most ministerial departments have been moved to this system or are in the process of being moved. Fifteen of the 21 reviewed ministerial departments, a complete list of which can be found in Table A5 in the appendix, use the overall search system and hence do not present any facets or filters to the searcher. The Department of Environment, Food and Rural Affairs9 does not currently provide a search system as it is also being moved to the GOV.UK single government website. Although using a different search system, the Prime Ministers web presence10 provides neither facets nor filters. Only four departments remain that feature facets or filters. Only the Department of Education11 provides document type and subject as facets, while also allowing for filtering searches by time frame and audience. The remaining three departments provide between one and four filter categories each, in total six - the number in brackets denotes the total number of departments employing a filter: audience (1), date modified (1), document type (2), organization (1), time frame (1), and topic ? subject (1). Table A6 in the appendix outlines the form of use and the resulting scoring for each facet/filter category in more detail. 7 https://www.gov.uk/ 8 https://www.gov.uk/search?q=test 9 http://www.defra.gov.uk/ 10 http://www.number10.gov.uk/ 11 http://www.education.gov.uk/search 27 Use of facets and filters in the web presence of the U.S. Government. No standards for the online content or search systems related to the content were found in the U.S. context; hence indications can only be drawn from the actual facet and filter features provided in search user interfaces. The USA.GOV search system12 provides filter possibilities for a limited number of document types and branches. 5 of the 18 departments, a complete list of which can be found in Table A7 in the appendix, reviewed use the USA.GOV search system. Only the Department of Labor13 does not provide any form of facets or filters. There are four departments presenting facets to the searcher. The Department of Energy14 employs document type and subject facets, the Department of State15 uses facets creator (speaker), document type, location (about), subject, and time frame, and the Department of Veterans Affairs16 provides document type and sub organizations facets. The Government Printing Office?s Federal Digital System17 provides digital government publication to the public and employs the facets collection (type - aggregation level), date published (time frame as well), author (government), organization, person (about), location, and keyword to facilitate this process. Additionally, the three formerly mentioned departments and the remaining 8 departments in total make 8 filter categories available to searchers - the number in brackets denotes the total number of departments employing a filter: Audience (1), document type (10), file type (3), language (1), location (about) (1), location (available) (1), organization (agency, branches) (10), and subject (topic) (2). Document type and organization receive a score of 3 due to their prevalent use across departments. Table A8 in the appendix outlines the form of use and the resulting scoring for each facet/filter category in more detail. 12 http://search.usa.gov/search?utf8=%E2%9C%93&sc=0&query=&m=&embedded=&affiliate=usagov&filter=moderate&commit=Search 13 http://www.dot.gov/gsearch 14 http://energy.gov/search/site 15 http://search.state.gov/search?q=test&site=state_en_stategov&client=state_en_stategov&output=xml_no_dtd&proxystylesheet=state_en_stategov&filter=0&entqr=3&lr=lang_en&oe=utf8&ie=utf8&getfields=*&search-button=Search 16 http://www.index.va.gov/search/va/va_adv_search.jsp 17 http://www.gpo.gov/fdsys/search/search.action?na=&se=&sm=&flr=&ercode=&dateBrowse=&govAuthBrowse=&collection=&historical=false&st=content%3A&psh=&sbh=&tfh=&originalSearch= 28 2.2.3 Facets and filter categories in the library and information domain. This section covers four library and information organizations. The organizations were deliberately selected to cover different perspectives. WorldCat18 covers a broad view on the potential characteristics of content in this domain. The OPACs of the Library of Congress19, the New York Public Library20, and the Library of the University of British Columbia21 cover the domain from a national, public and academic perspective, respectively. Table 2 provides a concise overview of facets or filter categories can be considered more or less important, while the following sub sections outline the major findings. It should be noted that every facet and filter category found in each of the four different systems was applicable across the entirety of the respective system, hence only the score 3 has been assigned. Table 2 Systems Review - Assessment of Use of Facets and Filter Categories in the Library and Information Domain Facet/Filter Category WorldCat LoC NYPL UBCL Total Also referred to as Creator 3 3 3 3 12 Author Date - published 3 3 3 3 12 New Releases Language 3 3 3 3 12 Subject 3 3 3 3 12 Topic Availability 3 3 3 9 no, yes; Location available; Item - Format 3 3 3 9 Binding Relation 3 3 3 9 Editions, Journal Title, Series, Volume, Issue Type - Document 3 3 3 9 Content Type, Document Type, Genre Audience 3 3 6 Accessibility 3 3 6 In Print, Online, In-Library Use Only, Take-Home-Rental Contributor 3 3 6 Coverage - Spatial 3 3 6 Publisher 3 3 6 Publication Venue Source 3 3 6 Journal Source Date - acquired 3 3 Organization 3 3 Special Attributes 3 3 Awards, List, Tag, User Type - Aggregation 3 3 Collection Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. The maximum score for a facet or filter category across all library and information organizations reviewed is 12. Abbreviations: LoC = Library of Congress; NYPL = New York Public Library; UBCL= Library of the University of British Columbia Status as of April 22nd, 2013 Table 2: Systems Review - Assessment of Use of Facets and Filter Categories in the Library and Information Domain 18 http://www.worldcat.org/ 19 http://www.loc.gov/index.html 20 http://www.nypl.org/ 21 http://www.library.ubc.ca/ 29 Use of facets and filters in WorldCat. WorldCat provides a simple and advanced search interface, as well as a faceted navigation component upon submitting a search query. The simple search interface allows for filtering by format only. Accession number, audience, author, content type (type of document), format, ISBN, ISSN, journal source, keyword, language, publication date, subject and title can be selected as fields or filters in the advance search interface. Identifiers, keywords and title are not considered as potential facets due their characteristics as outlined 2.2.2. The faceted navigation component made available to searchers next to the list of results contains the facets audience, author, content type (type of document), format, language, publication date, topic (subject). The results list also allows for viewing available editions and formats, and if applicable other titles in the series. Use of facets and filters in the Library of Congress OPAC. The Library of Congress OPAC provides a concise search system with a simple search and a faceted navigation upon submitting a search query. The simple search allows for filtering by file format. A faceted navigation interface supports searchers browsing through query results by making available the facets author (creator), availability status (online or not), format (original vs. online), language, location (spatial coverage), publication date, site (organization), and subject. Use of facets and filters in the New York Public Library OPAC. The New York Public Library uses Bibliocommons as OPAC which provides a simple search, an advanced search, a Classic Catalog, and upon submitting a search query a faceted navigation. The simple search allows for filtering by author, keyword, subject and title, as well as the special attributes list, tag, and user. As in earlier cases, keyword and title are not considered as potential facets. The Classic Catalog provides filters by call number, collection (Type ? Aggregation), and journal title. The call number, being an identifier for a particular item, is not considered as potential facet. Advanced search permits filtering by audience, availability location, award, author or contributor, collection (Type ? Aggregation), format, genre and content type (document type), geographic region (spatial coverage), language, publication date, publisher, series and subject. Upon submitting a search request accessibility (only used in library, online, 30 take home), acquisition date, audience, author, availability by location, format, genre, language, publication date, region (spatial coverage), tags, and topic (subject) are presented as facet. Use of facets and filters in the Library of the University of British Columbia OPAC. The OPAC of the Library of the University of British Columbia consists of four components, Summon ILS for general search, the Library Catalog for books and media, Indexes & Databases search, and Journal Search. The simple search only allows for keyword search by selected system, hence does not contribute any indication for potential facets. Each of the system components provides an advanced search which is briefly summarized in the following: ? The advanced search in Summons provides filter possibilities by accessibility (full text online, print material in library, scholarly peer-reviewed material), author, genre, issue, publication date, publication venue, and volume. It also presents filter possibilities by identifier (ISBN, ISSN) and title. Upon submitting a search request in Summons a faceted navigation component provides facet labels by availability (location), content type, language, publication date, and subject. ? The advanced search in the Library Catalog provides filter capabilities by author, corporate name/conference name (source), date published, format, language, location (availability), publisher, series, subject, and type. It also provides title and identifier (ISBN/ISSN) as filter. Upon submission of a search request to the catalog the result list is only accompanied by the filter categories availability location, format, and language. ? Using the advanced search for Indexes & Databases allows filtering by format and browsing by subject. It also presents filter possibilities by keyword and title. ? The advanced search for journals allows for browsing by subject and filtering by content type upon successfully submitting a search request. It also provides title and identifier (ISSN) as filter. 31 2.2.4 Facets and filter categories in commercial domain Four commercial web platforms have been selected based on the Alexa22 ranking for the Canadian top sites in business and economy to cover popular commercial systems. Out of the top five AbeBooks23, Amazon Canada24, eBay Canada25 and Kijiji26 have been selected. Table 3 provides a concise overview about which facets or filter categories can be considered more or less important, while the following sub sections outline the major findings. Table 3 Systems Review - Assessment of Use of Facets and Filter Categories in the Commercial Domain Facet/Filter Category AbeBooks Amazon.ca eBay.ca Kijiji Total Also referred to as Price 3 3 3 3 12 Price Range Terms and Conditions 3 3 3 9 Buying Format, Discount, Shipping and Payment Terms, Shipping Terms Organization 3 3 3 9 Department(Commerce), Model, Sub-Category, Store, Type, Type of Product Availability 3 3 3 9 no, yes; location; City, Province, Region, Territory, Town; distance to location with availability Rating - Source 3 2 3 8 Seller Rating, Seller Status Item - Format 3 3 2 8 Binding, Color, Composition, Lining, Material, Style, File - Format, File ? Medium, File Type, Protocol Type - Item 2 2 3 7 Document Type, Resource Type, Genre, Object Source 3 3 6 Brand, Seller Status 3 3 6 Authenticity, Condition, Grade, Grading, Original or Reproduction Special Attributes 3 1 1 5 Awards, Bullion, Circulation Status, Collectible Attributes Creator 3 1 1 5 Artist, Author, Maker, Manufacturer, Owner and Producer Date - published 3 1 1 5 New Releases Item ? Extent 2 2 4 Amount, Capacity, Diameter, Duration, File - Extent, Size, Speed, Width Subject 3 1 4 Athlete(about), Person(about), Sport(about), Team (about), Topic Location - Source 3 3 Location of Seller Type - Service 3 3 Relation 1 1 2 Character Family, Operating System, Platform, Series Audience 1 1 2 Age Range, Content Parental Rating, Gender, Weight of User Rating - Item 2 2 Coverage - Spatial 1 1 Destination Date - available 1 1 Date of Event Function 1 1 Movement, Resistance Language 1 1 Rights - Access Rights 1 1 Certification Time Frame 1 1 Coverage - Temporal Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. The maximum score for a facet or filter category across all library and information organizations reviewed is 12. Status as of April 23rd, 2013 Table 3: Systems Review - Assessment of Use of Facets and Filter Categories in the Commercial Domain 22 Based on http://www.alexa.com/topsites/category/Top/Regional/North_America/Canada/Business_and_Economy (Accessed: Apr 23 2013) 23 http://www.abebooks.com/ (Accessed: Apr 23 2013) 24 http://www.amazon.ca/ (Accessed: Apr 23 2013) 25 http://www.ebay.ca/ (Accessed: Apr 23 2013) 26 http://www.kijiji.ca/ (Accessed: Apr 23 2013) 32 AbeBooks. AbeBooks provides a very concise search system with a simple search, advanced search, and upon submitting a search request a faceted navigation. The simple search allows for filtering by author and browsing by subject, not considering limiting search to certain title elements, keywords or identifiers. The advanced search adds filters for binding, price range, publication date, and special attributes. The faceted navigation presents binding, collectible attributes, condition (new, used), seller location, seller rating, and shipping terms as facets. As all filters and facets can be used across the entire search system all receive a score of 3. Amazon Canada. Amazon Canada provides a usual search term based query as entry point as well as filtering by department (Type ? Category). In both cases the relevant sub departments are displayed as facets. Navigating through the levels of departmental hierarchies results in the display of more product-specific facets. In total 11 departments have been reviewed, a complete list of which can be found in Table A9 in the appendix. In addition to the department facet, a set of 16 facets can be identified based on specific facets in each department. Table 4 summarizes their occurrence and Table A10 in the appendix provides details which facet is being used in which department. Table 4 Systems Review - Occurrence of and Scores for Facets and Filters on Amazon.ca Facet/Filter Category Occurrence Score Facet/Filter Category Occurrence Score Audience 2 1 Organization 11 3 Availability 7 3 Rating ? Source 5 2 Creator 1 1 Rating ? Item 5 2 Date ? published 2 1 Relation 2 1 Function 1 1 Source 7 3 Item ? Extent 5 2 Special Attributes 2 1 Item ? Format 6 3 Terms & Conditions 10 3 Language 1 1 Type - Item 3 2 Price 8 3 Note: Number of departments n = 11; Status as of April 23rd, 2013 Table 4: Systems Review - Occurrence of and Scores for Facets and Filters on Amazon.ca 33 eBay Canada. The eBay search interface provides an initial possibility to filter a search by product category while providing specific facets after a search query has been submitted or the product hierarchy has been used to navigate to a sufficiently specific product category. All reviewed 32 departments/product categories, a complete list of which can be found in Table A11 in the appendix, feature product buying formats (auction, buy it now), condition, location, price range, seller rating status, and shipping and payment terms as facets, as well as allow for defining a price range filter by means of text input fields. In addition, most of the departments provide more specific facets according to their product categories. Table 5 summarizes the occurrence of facets amongst the reviewed departments. Table A12 in the appendix outlines the use of facets by department in more detail. Table 5 Systems Review - Occurrence of and Scores for Facets and Filters on eBay.ca Facet/Filter Category Occurrence Score Facet/Filter Category Occurrence Score Audience 5 1 Rating - Source 32 3 Availability 32 3 Relation 5 1 Coverage Spatial 1 1 Rights - Access Rights 2 1 Creator 3 1 Source 16 3 Date ? available 1 1 Special Attributes 1 1 Date ? published 3 1 Status 32 3 Item ? Extent 7 2 Subject 6 1 Item ? Format 9 2 Terms and Conditions 32 3 Price 32 3 Time Frame 1 1 Organization 32 3 Type - Item 14 2 Note: Number of departments n = 32; Status as of April 23rd, 2013 Table 5: Systems Review - Occurrence of and Scores for Facets and Filters on eBay.ca Kijiji. Kijiji provides an initial search interface with filter categories and a faceted navigation component that appears after selecting a filter category or entering a search term. The initial interface shows three filters. The first one is for products (type of category) or services (type of services), such as cars & vehicles or jobs & services, respectively. The second filter, Community Pages, provides a set of different type of documents, such as resumes, and types of categories, such as occupations. The third filter is a location filter for availability of products or services by province/territory, area, and city/town. After selecting a category or submitting a search query a faceted navigation interface appears next to the results list. In addition to category and availability by location the faceted navigation provides a distance slider (a form of availability by 34 location), a facet indicating whether a classified aims a buying or selling (type of document) and the facet range for price. 2.2.5 Summary and implications for study 1. The use of facets and filters in 12 search interfaces used in government, library and commercial domains was reviewed to understand the use of content characteristics and their prevalence. Combining and aggregating the score results of the three domains resulted in a set of 45 facets and filter categories. This set was used for determining the content characteristics to be included in the questionnaire used in study 1. 11 of the 45 content characteristics were used across all three domains with widely differing levels of prevalence. 19 of the 45 content characterstics were used in only one of the domains examined showing a relatively low level of prevalence with the exception of characteristics only used in the commercial domain, such as price. The inclusion of a content characteristic in the questionnaire conducted as part of study 1 was based on the scores as well as comprehensibility of and assumptions made about the concepts. As outlined in 2.1.10 it was also necessary to select facets and filter categories with different indications about their level of importance, hence with high, medium and low scores. The 14 mostly clearly comprehensible facet concepts were selected for inclusion in the questionnaire, some of which are representing very similar concepts, such as being about a specific geographical entity and being physically located in a specific geographical entity. While Table A13 in the appendix provides more details on the included facets, Table A14 summarizes the facets and filter categories not selected. These are the facets selected for inclusion in the questionnaire: ? With a high score: Item ? Format, Type ? Item, Availability, Date ? published ? With medium score: Audience, Coverage ? Spatial, Terms and Conditions, Type ? Category, Rating ? Source, Time Frame ? With a low score: Item ? Extent, Location ? Source, Date ? available, Rating ? Item 35 3 Study 1: Task-based Perceived Usefulness of Facets 3.1 Research Design 3.1.1 Introduction. The purpose of this study was to establish an indication of perceived usefulness of facets considering types of search tasks. This was done by means of a questionnaire-based online survey asking participants to rate the usefulness of facets when considering different types of search tasks. The expectation for the outcome of this phase of the research was contributing to answering research question 1 by determining the perception of facet usefulness overall and by type of task. In preparation for this study the list of types of search tasks based on the literature review was used. The number of types was limited to three, Doing, Known-Item, and Learning, covering the different aspects of types of tasks, such as specific versus general. The selection of facets to be included was based on the performed systems review and included 14 content characteristics, facets, with different levels of indicated importance as outlined in 2.2.5. 3.1.2 Recruitment and participants. Summary. Being part of the e-informing the public project, participation was limited to citizens and permanent residents of Canada aged 22 or older (Freund & Berzowska, 2010). This was done to make results, particualarly related to the e-government domain, more easily comparable to previous studies in this research project. Originally participant recruitment was planned to be conducted only via posters in public places in Vancouver, such as bulletin boards at libraries or community centres, and Facebook advertisements. However, these approaches proved to be inadequate as only 1 participant was recruited via Facebook ads and none via posters in public places. Hence additional recruitment avenues were added subsequently. These included posting of recruitment notifications to mailing lists of university departments at the University of British Columbia and professional and research associations, as well as postings to groups on the social networks Facebook and LinkedIn. Recruitment using all avenues was conducted over a period of 6 weeks from the end of February to early April 2013. In total 94 responses were received of which 11 were discarded. Of the 36 remaining 83 responses 65 are considered entirely completed while 18 cover at least one completed assessment of all characteristics for one of the three types of search tasks. Basic participant information provided within the completed questionnaire submissions are summarized below, while details can be found in Appendix B: ? Age groups: 46.2% 22-31, 30.8% 32-41, 16.9% 42-51, 6.2% 52 or older ? Gender: 60.0% female, 35.4% male ? Status: 47.7% full time students, 4.6% part time students, 47.7% non-students ? Employment status: 43.1% working full time, 32.3% working part time ? Highest degree earned or in progress: 66.2% Master, 18.5% doctorate, 10.8% Bachelor ? Self-reported skill level in searching the Internet: On a scale from 1(low) to 7(high) 96.9% of participants rated themselves as 5 or higher. Representativeness of sample. Comparing the sample of 65 participants having provided basic information with statistics provided by Statistics Canada, the presence of significant skews for age, student status, employment status, and highest degree earned or in progress, as well as a somewhat less significant skew for gender can be observed. The relative frequency of the two youngest age groups of participants with ages ranging from 22 to 41 is 77.0% while this age group is only represented 36.7% in the entire population27. Female participants are overrepresented with 60% relative frequency while accounting for approximately 51% of the entire population28. While an exact comparison of sample and population cannot be conducted for student status, employment status, and highest degree earned or in progress due to the selection of age groups in this study and a missing granularity in the available Statistics Canada data29, indications regarding representativeness can be derived. Within the sample 66.7% out of the participants in 27 Compare Tables A15 and A22 in Appendix B and C, respectively. 28 Compare Tables A16 and A22 in Appendix B and C, respectively. 29 Statistics Canada provides data for employment status, and highest degree earned or in progress for the age group of 15 years and over, and sub groups thereof that do not match the groups used in this study. Statistics Canada provides data for student status for the age group of 15 to 29 years, and sub groups thereof that do not match the groups used in this study. Employment status and highest degree earned or in progress are compared using the entirety of the sample, while only the age group of 22 to 31 years is used to compare the student status. With this limitation in the comparison in mind it should still be possible to derive indications for representativeness. 37 the age group from 22 to 31 years are full-time students, while 30% of this age group are not currently undertaking a formal study program. In the closest possible comparison data provided by Statistics Canada, approximately 44.3% of the population aged 15 to 29 years are undertaking full-time studies, while approximately 54.4% can be considered as not studying. This indicates that full-time students are overrepresented in the sample30. The group of full-time employees seems to be underrepresented in the sample as approximately 75.3% of the population aged 15 or above are full-time employees while only 43.1% of the study participants consider themselves full-time employees31. Hence, the group of part-time employees can be seen as overrepresented in the sample. 95.5% of study participants had at least a Bachelor?s degree or were pursuing one while this only applies to approximately 18.1% of the population aged 15 or above32. While it is obvious this is a sample of convenience, not all of these overrepresentations necessarily affect the results of this study. Recruitment approach. The initially selected recruitment via Facebook Ads was unsuccessful; hence it is warranted to briefly look at the broader issue of subject recruitment. Subject recruitment and its associated compensation for participants constitute the main logistical challenge for experimental research in the social sciences (Samuels & Zucco, 2012). While a more detailed discussion of the pros and cons of different recruitment and compensation methods might be merited, particularly in the context of new possibilities provided by social networks, this is not part of this research project. For the purpose of this study, the recruitment and compensation approach used by Samuels and Zucco (2012) has been emulated for the most part targeting residents of Canada aged 22 or older. Their Facebook Ad consisted of the description ?Win an iPad2! University researchers want your opinion. Fill out a ten-minute questionnaire and you?re eligible to win an iPad2 (1 in 3000 chance).? and an image showing an iPad2. As the sample size in this study was significantly lower, a planned number of 200 participants, the prize was reduced in value to a $50 gift certificate while the text of the ad was phrased similarly. Pilot ads were published on Facebook for 7 days, but only resulted in very few clicks and no activity beyond the first page of the online questionnaire. At the same time a pilot with a poster notice did not result in any activity, while a 30 Compare Tables A21 and A24 in Appendix B and C, respectively. 31 Compare Tables A18 and A25 in Appendix B and C, respectively. 32 Compare Tables A19 and A23 in Appendix B and C, respectively. 38 pilot using a mailing list for the distribution of an invitation to participate in the study resulted in 5 clicks with 3 completed responses. Hence several changes to the recruitment and compensation approach were made. As mailing lists seemed to work best, it was decided to use this recruitment venue as well as posting the study invitation in relevant groups on the social networks Facebook and LinkedIn. The prize of the draw was changed to a Samsung Galaxy Tablet (7in) ? a four-fold increase in value. The first page of the questionnaire featuring a summary of the information enabling informed consent was also revised and shortened. Successively adding poster notices, and distributing invitations via mailing list and relevant groups, 94 responses were collected. Only one of them was a result of a click on a Facebook Ad, although 130 clicks on the Facebook Ads were recorded. The reason for the success and failure of different recruitment venues might be related to the characteristics of the venue and incentive. While there have been studies successfully using Facebook Ads for recruitment beyond the study by Samuels and Zucco (2012) (e.g. Gerben, 2010), there seem to be indications that ads in social networks might be not very effective depending on the aim of the ad and on the geographic area targeted. For example, Worstall (2012) elaborates on whether Facebook Ads are effective for click-throughs or brand building. Holiday (2012) more vehemently questions the effectiveness of Facebook Ads and asserts that they are part of ?Facebook Ponzi Scheme? in which users are led to believe that with more experimentation their ads might bring them the click through rate and outcomes they aim for. But this experimentation comes with a cost. In an experiment conducted by the BBC it was indicated that advertising on Facebook might be more or less effective depending on the geographic area targeted and that they are particularly ineffective in mature advertisement markets, such as the UK and U.S., and by inference Canada (Cellan-Jones, 2012). Considering incentives for study participation, Dillman, Smyth, and Christian (2009) suggested that lotteries and prize draws are very ineffective and providing each participant with a little material incentive after completing the questionnaire is a bit more effective, while providing participants with a small prepaid token of trust in form of $1 to $10 upfront seems to be most effective. However prepaid incentives are impractical when it comes to questionnaire-based online surveys and Dillman et al. (2009) refer to recruiting participants only via email and the sources supporting the claims made for the effectiveness of incentives are based on sources that are two decades old and which mainly deal with mail surveys (Church, 1993; James & Bolstein, 39 1990, 1992; Johnson & McLaughlin, 1990). Hence any impacts by technological developments over the last two decades are not considered in this argument and at least some of the assumptions underlying the claims could be invalid by now. For example, Cobanoglu and Cobanoglu (2003) found no significant increase in response rates by using prize draws, but their findings indicate that offering respondents the chance to enter a draw for a bigger prize results in the highest response rate of incentives assessed for internet-based surveys. 3.1.3 Design of data collection instrument. A self-guided online questionnaire was employed as the means of data collection. It consisted of three separate components, an instruction/consent page also providing images introducing the concept of faceted search, the main questionnaire, and a page providing the participant with the opportunity to enter the draw and to indicate their interest to participate in study 2. The main questionnaire consisted of one page per type of search task (Doing, Known-Item, and Learning) for the participant to assess the usefulness of the selected 14 facets, a page to provide additional comments to specific statements or questions, and a page with questions regarding basic information which was placed at the end on purpose to conform with Dillman et al.?s (2009) Guideline 6.3 to place more sensitive questions near the end of the questionnaire. Each type of task was accompanied by an abstract description and three real-world example scenarios as suggested by Freund and Nilsen (2008), one from the e-government domain, one from the library domain and one from the commercial domain. The example scenarios are summaries of the scenarios used in the study conducted by Freund and Berzowska (2010) for the e-government domain. For the library and commercial domains the example scenarios have been designed in alignment to the wording and structure of the scenarios in the e-government domain. The assessment of usefulness of facets was performed on a 7-point Likert scale (1 = Low to 7 = High), similarly to a previous study conducted as part of the e-informing the public project (Freund & Berzowska, 2010). For the purpose of this study usefulness is considered to be ?the extent to which information objects are suited to the users? tasks and goals? (Freund, 2011, p.1). After assessing all task types, participants also had the opportunity: ? To indicate whether they would find other facets useful based on a list of facets not included in the questionnaire and by only indicating yes or no. ? To indicate any other facets they can think of being useful by means of comment box. 40 ? To provide their impressions about search engines using facets and when facets are useful. ? To provide comments about the questionnaire, particularly challenges they encountered in completing it. To account for order effects multiple version of the questionnaire were created featuring a different order for the pages containing the usefulness assessment of facets for different types of search tasks. The different versions of the questionnaire were hosted in Canada using the UBC IT Survey Tool Enterprise Feedback Management (EFM) to address privacy legislation and concerns. Unique links to different versions of the questionnaire were used for different recruitment venues. On average it took participants 13 minutes to complete the questionnaire. A complete version of the questionnaire can be found in Appendix D. 3.1.4 Data analysis. The questionnaire responses provide quantitative data for basic participant information questions and Likert scale responses. A limited set of qualitative data consisting of the comments participants provided at the end of the questionnaire was also collected. Data was analyzed using SPSS 21. Search tasks types constituted the independent variable and the usefulness assessments for the 14 different facets served as the dependent variable. Hence, the determination of usefulness of facets per type of search task and the comparison across types of search tasks constituted the main analysis activities. The data for the assessed usefulness of facets was analysed using descriptive statistics to establish an initial list of more or less useful facets per task type, based on participants? perceptions. Subsequent Kruskal-Wallis and Mann-Whitney U tests were then performed to test for statistically significant differences in perceived usefulness of facets by task type. The data analysis concludes with the main themes that can be found in the qualitative comments provided by participants. Before conducting the data analysis the data was examined to identify invalid and/or incomplete responses. In total 11 out of the 94 received responses were removed. Seven of the 11 removed responses were entirely empty, 2 of the 11 were exactly the same and only alternated between using 1 and 7 for assessing the usefulness of facets, and 1 of only rated the usefulness of two facets relating to one type of search task. Another response was removed due to the participant not being a citizen or a permanent resident of Canada. 41 Reviewing the assumptions underlying statistical variance tests, the collected data set was tested via non-parametric Kruskal-Wallis and Mann-Whitney U tests. The alternative, one-way ANOVA tests, are based on six assumptions about the data to be analyzed (Lund Research, 2013b). At least one of them, the assumption that the ?dependent variable should be approximately normally distributed for each category of the independent variable?, was not met. Perceived usefulness scores were not normally distributed for the three different types of search tasks Doing, Known-Item, and Learning, as assessed by Shapiro-Wilk's test (p > .05)33. As the dependent variable, Perceived Usefulness, is measured at the ordinal or interval/ratio level and the independent variable, the Type of Search Task, consists of two or more categorical and independent groups non-parametric variance tests can be used (Lund Research, 2013a). A Kruskal-Wallis test was applied using all three types of tasks while the Mann-Whitney U test was used for pairwise comparisons. 3.2 Results 3.2.1 Level of perceived usefulness of facets by type or task. Participant?s perceptions of usefulness of facets were quite consistent across tasks. As can be seen in tables 6, 7, 8 and 934, the order of facets is very similar across all types of search task when they are ranked based on the arithmetic mean of perceived usefulness. The facets Date created or published, Department (Organization), Geographical area (about), Timeframe, and Type of document seem to be considered most useful independent of the task context. In contrast, the facets Ratings of provider, Audience, and Terms of use seem to be considered as least useful. The only differences amongst the perceived usefulness of these facets are rank order and lower means in the context of the Doing task type. 33 More details can be found in Table A27 in the appendix 34 More comprehensive descriptive statistics can be found in Table A26 in the appendix. 42 Table 6 Study 1 - Mean Perceived Usefulness of Facets Across Task Types Facet Mean Perceived Usefulness N Department (Organization) 5.41 82 Type of document 5.36 83 Date created or published 5.31 83 Geographical area (about) 5.22 83 Timeframe 5.17 83 Format of object 4.82 83 Availability 4.49 83 Ratings of object 4.46 82 Size 4.45 83 Date available 4.44 83 Geographical area (location) 4.36 83 Ratings of provider 4.33 83 Audience 3.85 83 Terms of use 3.36 83 Mean sorted in descending order. Table 6: Study 1 - Mean Perceived Usefulness of Facets Across Task Types Table 7 Study 1 - Mean Perceived Usefulness of Facets for Doing Task Type Facet Mean Perceived Usefulness N Type of document 5.32 76 Department (Organization) 5.20 75 Geographical area (about) 4.96 75 Timeframe 4.87 75 Date created or published 4.83 75 Format of object 4.76 76 Ratings of object 4.57 72 Availability 4.57 76 Ratings of provider 4.35 75 Size 4.29 75 Geographical area (location) 4.26 76 Audience 4.17 76 Date available 4.11 74 Terms of use 3.45 76 Mean sorted in descending order. Table 7: Study 1 - Mean Perceived Usefulness of Facets for Doing Task Type Table 8 Study 1 - Mean Perceived Usefulness of Facets for Known-item Task Type Facet Mean Perceived Usefulness N Department (Organization) 5.68 69 Type of document 5.49 69 Date created or published 5.46 69 Geographical area (about) 5.31 67 Timeframe 5.14 69 Date available 4.97 69 Format of object 4.79 67 Size 4.45 69 Availability 4.43 69 Geographical area (location) 4.38 69 Ratings of provider 4.32 69 Ratings of object 4.30 69 Audience 3.53 70 Terms of use 3.19 69 Mean sorted in descending order. Table 8: Study 1 - Mean Perceived Usefulness of Facets for Known-item Task Type Table 9 Study 1 - Mean Perceived Usefulness of Facets for Learning Task Type Facet Mean Perceived Usefulness N Timeframe 5.55 74 Date created or published 5.53 75 Geographical area (about) 5.49 75 Department (Organization). 5.45 73 Type of document 5.40 75 Format of object 4.79 75 Date available 4.66 74 Size 4.58 74 Ratings of object 4.58 74 Availability 4.56 75 Geographical area (location) 4.47 75 Ratings of provider 4.35 75 Audience 3.93 75 Terms of use 3.42 74 Mean sorted in descending order. Table 9: Study 1 - Mean Perceived Usefulness of Facets for Learning Task Type 43 3.2.2 Analysis of variance. A Kruskal-Wallis test was run to determine if there were differences in Perceived Usefulness scores between Types of Search Tasks. Pairwise comparisons were performed using Dunn's (1964) procedure with a Bonferroni correction for multiple comparisons. A significant difference in Perceived Usefulness scores was observed between the different Types of Search Tasks, ?2(2) = 8.375, p = .013. Perceived Usefulness scores were significantly different between Doing (Mean = 4.55) and Learning (Mean = 4.77) (p = .013) but not between any other combinations35. For each content characteristic a Kruskal-Wallis test was run to determine if there were differences in Perceived Usefulness scores between Types of Search Tasks. Pairwise comparisons were performed using Dunn's (1964) procedure with a Bonferroni correction for multiple comparisons. Between the different types of tasks four content characteristics showed statistically significant differences: Date available (?2(2) = 7.507, p = .023), Date created or published (?2(2) = 6.039, p = .049), Geographic Location (about) (?2(2) = 7.632, p = .022), and Timeframe (?2(2) = 8.054, p = .018). Post hoc analyses revealed the following: ? Date available: Statistically significant difference between Doing (Mean = 4.11) and Known-Item (Mean = 4.97) (p = .021) but not between any other combinations36. ? Date created or published: No statistically significant differences in Perceived Usefulness score for any combination of types of tasks37. ? Geographic location (about): Statistically significant difference between Doing (Mean = 4.26) and Learning (Mean = 5.49) (p = .020) but not between any other combinations38. ? Timeframe: Statistically significant difference between Doing (Mean = 4.87) and Learning (Mean = 5.55) (p = .014) but not between any other combinations39. 3.2.3 Comments by participants. Comments by participants echoed some of the challenges faced when using faceted systems as discussed earlier. Nine participants mentioned that they are concerned ?that what you're looking for may be weeded out of the search?(P80) and that ?[f]ilters are largely useful only in so much as the information they are referencing is properly tagged/searched\"(P82). Overall participants 35 More details can be found in Table A28 in the appendix 36 More details can be found in Table A29 in the appendix 37 More details can be found in Table A30 in the appendix 38 More details can be found in Table A31 in the appendix 39 More details can be found in Table A32 in the appendix 44 also stated that it is likely that they would rate the usefulness of content characteristics differently when faced with different situations, while one participant stated that the perceived usefulness of content characteristics is more influenced by personal behaviour. Twelve participants indicated that the ?usefulness [of content characteristics] is highly dependent on the specific scenario?(P45) and that the usefulness ?of the filters [?] differs GREATLY depending on the specific information?(P64) searched for. One the other hand at least one participant stated: \"I think what filter categories you use may be a matter of personal information seeking behaviour. Although the scenarios presented were different, I found I kept choosing the same ones\"(P68). 3.3 Summary The findings of this study indicate that there is a high similarity in the perceived usefulness of content characteristics/facets for all three types of search tasks investigated. There also is a statistically significant difference between the Doing task type and the Learning task type, and in case of the content characteristic Date available also between the Doing task type and the Known-Item task type. The differences are primarily related to the lower mean scores in perceived usefulness of content characteristics for the Doing task type. While there is some variation in the rankings of content characteristics across task types, their perceived usefulness is generally consistent. Consequently, the findings of this study do not support the concept of changing the facet hierarchies in search user interfaces for different types of tasks. 45 4 Study 2: Task-based Use of Facets 4.1 Research Design 4.1.1 Introduction. The purpose of this study was to investigate the actual use of facets for different types of search tasks and to compare the performance of search systems with facets and without facets. The study determined the level of satisfaction, effectiveness and efficiency provided by a search system without facets and the same search system with facets considering types of search tasks. Originally it was planned to use an additional search system that provided different sets of facets depending on the type of search task performed. However, as study 1 indicates that the most useful facets are perceived to be the same across types of search tasks, there is no basis for using task-dependent sets of facets. The research was conducted via a between-subject experimental user study employing the Faceted Retrieval of E-government Documents (FRED) search system40 () in two different states, without facets, referred to as the baseline system, and with facets, referred to as the experimental system. The expected outcome of this study is three-fold: ? Contributing to answering research question 1 by testing the results of the perceived usefulness of facets identified in phase 1 by observing actual use. ? Contributing to answering research question 2 by assessing whether a faceted search system is more useful for discovering online content than a search system without facets. ? Testing hypotheses. 4.1.2 Recruitment and participants. Summary. Being part of the e-informing the public project, participation was limited to citizens and permanent residents of Canada aged 22 or older (Freund & Berzowska, 2010). Participant recruitment was conducted via mailing lists of university departments and student societies at the University of British Columbia, as well as through a participant recruitment system provided by the Department of Computer Science at the University of British Columbia. Participants of study 40 http://www.diigubc.ca/fred 46 2 having indicated interest in participating in further research were also invited. Using these recruitment venues, this sample can be considered one of convenience. After conducting an initial pilot, recruitment was carried out over a period of 4 days in mid-May 2013. 22 experimental sessions were conducted from May 20th to 27th 2013, one of which was a second pilot. Of the 21 non-pilot sessions one was discarded due to the participant specifically stating that his search behaviour is completely different from the behaviour exhibited during the session. Each participant received an honorarium of $20 at the end of the session. The basic information of the 20 participants41 considered for analysis are summarized below, while details can be found in Appendix F: ? Age groups: 15 participants in age group 22-31, 4 participants in age group 32-41, 1 participant in age group 42-51 ? Gender: 13 female participants, 7 male participants ? Student status: 15 full time students, 2 part time students ? Employment status: 1 participant working full time, 11 participants working part time ? Highest degree earned or in progress: All participants are pursuing or have completed a Bachelor?s degree or higher ? Self-reported skill level in searching the Internet: On a scale from 1(low) to 7(high) 19 participants rated themselves as 5 or higher Representativeness of sample. Comparing the sample of 20 participants with population statistics provided by Statistics Canada, the presence of significant skews for age, student status, employment status, highest degree earned or in progress, and gender can be observed. The relative frequency of the two youngest age groups of participants ranging from 22 to 41 years is 95.0% (19 of 20 participants) while this age group is only represented 36.7% in the entire population42. Female participants are 41 According to Nielsen (2006) 20 participants can be considered as a sufficiently big sample for offering a ?reasonably tight confidence interval? for usability metrics. 42 Compare Tables A33 and A22 in Appendix B and C, respectively. 47 overrepresented with 65% relative frequency (13 of 20 participants) while accounting for approximately 51% of the entire population43. While an exact comparison of sample and population cannot be conducted for student status, employment status, and highest degree earned or in progress due to the selection of age groups in this study and a missing granularity in the available Statistics Canada data44, indications regarding representativeness can be derived. Within the sample 73.3% (11 out of 15 participants) out of the participants in the age group from 22 to 31 years are full-time students, while 13.3% of this group is not currently undertaking a formal study program. In the closest possible comparison data provided by Statistics Canada, approximately 44.3% of the population aged 15 to 29 years are undertaking full-time studies, while approximately 54.4% can be considered as not studying. This indicates that full-time students are overrepresented in the sample45. The group of full-time employees seems to be underrepresented in the sample as approximately 75.3% of the population aged 15 or above are full-time employees while only 5% (1 participant) of the study participants indicated to be a full-time employee46. Hence, the group of part-time employees can be seen as overrepresented in the sample. All study participants had at least a Bachelor?s degree or were currently pursuing one while this only applies to approximately 18.1% of the population aged 15 or above47. While this sample of convenience is not representative of the Canadian population as a whole, these discrepancies are not likely to affect the internal validity of findings, but do place some limits on generalizability. 4.1.3 Experimental system and tasks. Summary. The research was conducted via a between-subject experimental user study employing the FRED search system48. The domain of e-government was selected as it is expected to have a low level 43 Compare Tables A34 and A22 in Appendix B and C, respectively. 44 Statistics Canada provides data for employment status, and highest degree earned or in progress for the age group of 15 years and over, and sub groups thereof that do not match the groups used in this study. Statistics Canada provides data for student status for the age group of 15 to 29 years, and sub groups thereof that do not match the groups used in this study. Employment status and highest degree earned or in progress are compared using the entirety of the sample, while only the age group of 22 to 31 years is used to compare the student status. With this limitation in the comparison in mind it should still be possible to derive indications for representativeness. 45 Compare Tables A39 and A24 in Appendix B and C, respectively. 46 Compare Tables A36 and A25 in Appendix B and C, respectively. 47 Compare Tables A37 and A23 in Appendix B and C, respectively. 48 http://www.diigubc.ca/fred 48 of user expertise (Freund, 2008) and average citizens are usually overwhelmed by the amounts of electronic data governments provide to them (Fountain, 2001). Usability is important particularly when systems are targeted at an external audience (Buie & Murray, 2012), which should largely be the case in the e-government domain. It is also necessary to consider usability of search systems early and often (Brinck, Gergle, & Wood, 2002). Hence, this experimental user study employed usability measures. Each participant received a set of three scenario-based activities, one scenario from each type of search task, Doing, Known-Item, and Learning. The order of activities was rotated to account for order effects. Each participant performed the activities only using one version of the FRED system, the baseline or the experimental system. Participants using the experimental system were either using a system in which the facets were ordered from A to Z or vice versa49. Set Up of the Experimental System. The FRED system is an experimental system that employs a faceted, metadata-driven approach for accessing e-government content, particularly Government Canada content. It is based on Apache Nutch and Apache Solr software because of their robust integration of meta-data extraction and faceted search features. The same Government of Canada agencies and departments as reviewed in Chapter 2.2 were crawled and approximately 480,000 web pages were indexed in June 2012. While metadata is embedded in many of these pages, it was not applied consistently and nearly 100 metadata elements were found to be in use. Only 19 of these elements were used in more than 10% of the web pages (Freund, Jinglewski, & Kessler, 2012). To determine the usability of the metadata for the purpose of this study, the indexing process was performed a second time in February 2013 focusing on pages in the English language only. The resulting set of approximately 240,000 pages exhibited similar characteristics to the previous set in terms of metadata availability. Hence, it became obvious that the existing metadata needed to be supplemented in order for a metadata-based faceted navigation system to work properly and not weed out pages lacking metadata or using non-standard elements. As this was not possible for the entire set of 240,000 web pages in the time available, a smaller subset of documents and a limited set of facets were selected based on the scenarios to be used for participants? search 49 More details can be found in Table A40 in the appendix. 49 tasks. A small-scale version of FRED was created for the study with a manually tagged collection of approximately 1000 documents. Out of the 5 content characteristics indicated as most useful in study 1, 4 were selected to be included in study 2. These were Department (Organization), Type of document, Date created or published, and Geographical area (about). The content characteristic Timeframe was not included as it is very difficult to determine for government online content in most cases. Most content is covering the present until being replaced by new content. In addition, two content characteristics considered less useful were selected to be included, Audience and Size. These content characteristics were chosen to test perceptions versus actual use and based on previous research and the systems review, which indicated that they might be of higher usefulness in the e-Government domain. The content for these characteristics, if not available, was collected or created using the following means: ? For content characteristics Department (Organization), Geographical area (about) and Size: These were automatically generated based on URL, location entity extraction, and number of characters, respectively. ? For the other content characteristics: If not available they were manually tagged by members of the FRED research team using metadata schemes provided by Government Canada for Audience and Type of document50, and in case of Date created or published by dates indicated within the web pages. ? The facet labels for Department (Organization), Geographical area (about), and Type of document were organized in two-level hierarchies, which had been created for the FRED system based on card sorting experiments (Freund et al., 2013). The types of search tasks used in this study are the same as used in study 1, Doing, Known-Item, and Learning. The scenarios used as the basis for the user experiments were created based on the government domain examples used in study 1 where possible: Doing: An elderly uncle has had a stroke and is confined to a wheelchair, but he and your aunt want to continue to live in their own home. You are seeking information on how to adapt their home to the new circumstances. 50 See Treasury Board of Canada Secretariat (2012): http://www.tbs-sct.gc.ca/im-gi/imrc-crgi/metadata-metadonnees-eng.asp#s4 50 Known-Item: You are performing historical research into First Nations communities and are looking for records of individuals. You have heard that it is possible to obtain these records from a federal government agency. You are looking for the official document needed to send an information request to this agency. Learning: After listening to an interesting radio program about weather disasters, you want to learn more about the effects of extreme weather situations and their impact on different communities in Canada. You are seeking information to learn about this topic. Based on the developed scenarios a list of 1,005 unique web pages was identified by issuing different possible text queries against the FRED index. On a scale from 1 to 7 participants rated the realism of the scenarios between 4.9 and 5.65 on average for Known-Item and Doing scenarios, respectively, with the Learning scenario?s realism being rated with 5.4. On the other hand, on average participants rated their knowledge of the scenario topics lower, at 1.95 for the Known-Item scenario, 3.1 for the Doing scenario, and 2.85 for the Learning scenario. Procedures. Participants were asked to go through three components to complete the session. The first component consisted of a brief set of basic information questions similar to the set used in study 1. The second component consisted of three search tasks to be performed by the participant. Each task was preceded by providing the participant with a scenario description and a brief pre-task questionnaire asking about the scenario realism and prior knowledge about the scenario. Participants? interactions with the FRED system were observed and recorded using Morae. After each task participants were presented with a brief post-task questionnaire asking to assess their satisfaction, success, and new level of knowledge, and if applicable to provide comments on information and system features that might have been helpful in completing the task. After completion of all three tasks, participants were asked to assess their perception of usefulness and ease-of-use of the system, and provide comments on what kind of system features they found particularly useful and which system features not available would have been helpful. Participants 51 using the experimental system were also asked to assess the individual usefulness of the six facets51. Measures. The experiment tracked a number of measures to evaluate five variables: the perceived usefulness of facets, the actual use of facets, user satisfaction, efficiency and effectiveness. Measures were aggregated at the system state level, meaning that the baseline system was compared to the experimental system. Table 10 summarizes the variables and measures, and indicates which research questions and hypotheses they address. Measures referring to results are including all search results, both pages accessible directly from the search system?s results lists, and those accessed by following links from these pages. Where applicable additional measures only including results directly accessible from the search system?s results lists are included. Research questions: 1) Considering types of search tasks, is there a difference in the perceived usefulness of facets for discovery of online content? Does the actual use of facets in search systems vary by type of task? 2) Are static facets and dynamic task-dependent facets useful for discovery of online content? Hypotheses: 1) Search systems providing faceted search lead to a higher user satisfaction compared to search systems without this capability. 2) Search systems providing faceted search lead to a higher effectiveness compared to search systems without this capability. 3) Search systems providing faceted search lead to a higher efficiency compared to search systems without this capability. 51 Study protocol, pre-questionnaire, search instructions and questionnaires, and post-questionnaires can be found in appendices I to L, respectively. 52 Table 10 Study 2 - Summary of Variables and Measures Tracked Variable Definition Measures by Scenario and Facet Availability/System RQ H Perceived usefulness of facets* The level of perceived usefulness of a facet in the experimental system. Self-reported 7-point Likert scale of perceived usefulness by facet 1, 2 Actual use of facets* The level of use of a facet in the experimental system for a certain type of task. Frequency count: number of clicks per facet compared to the number of clicks used for other facets. 1, 2 Satisfaction The level of satisfaction by a user using the experimental system in a certain system state for a certain type of task. - Perceived Ease of Use: Self-reported 7-point Likert scale of perceived clearness of interaction, required mental effort and ease of use** - Self-reported 7-point Likert scale for level of satisfaction after each task - Self-reported 7-point Likert scale for level of challenge of searching after each task 1, 2 1 Effectiveness The level of success in retrieving the content meeting the information needs generated by a task. - Self-reported 7-point Likert scale of Perceived Usefulness of system** - Self-reported 7-point Likert scale of perceived success after each task - Perceived Knowledge Gain: Self-reported 7-point Likert scale of perceived degree of knowledge about topic after each task versus self-reported 7-point Likert scale of perceived degree of knowledge about topic before performing task - Number of results bookmarked - Relevance of results bookmarked*** 1, 2 3 Efficiency The level of effort required to complete task: least effort given compared to same outcome. - Task completion time (in seconds) - Number of text queries - Number of FRED result pages viewed - Number of facet uses - Number of facet filter uses - Number of results looked at - Number of results looked at per minute - Number of results looked at per text query - Number of results looked at per FRED result page viewed - Number of results looked at per facet use - Number of results looked at per facet filter use 1, 2 2 RQ = Research Question(s) addressed H = Hypothesis addressed Research questions: 1) Considering types of search tasks, is there a difference in the perceived usefulness of facets for discovery of online content? Does the actual use of facets in search systems vary by type of task? 2) Are static facets and dynamic task-dependent facets useful for discovery of online content? Hypotheses: 1) Search systems providing faceted search lead to a higher user satisfaction compared to search systems without this capability. 2) Search systems providing faceted search lead to a higher effectiveness compared to search systems without this capability. 3) Search systems providing faceted search lead to a higher efficiency compared to search systems without this capability. * Is only applicable for experimental system. ** Reporting only possible by facet availability/system, not by scenario. *** The relevance of the results bookmarked is determined in two ways: 1) Manual assessment of whether a result is not relevant, somewhat relevant, or very relevant. 2) Automatic assessment based on the number of times the result has been bookmarked in total across all participants. Table 10: Study 2 - Summary of Variables and Measures Tracked 4.1.4 Data analysis. The questionnaire responses provided quantitative data based on basic participant information, Likert scales, and interval and ratio measurements from logs of participants? interactions with the system. System interactions were recorded with Morae. Morae logs were analyzed manually to extract specific measures. A limited set of qualitative data consisting of the comments participants provided at the end of the questionnaire was also collected. To answer the research 53 questions and test the hypotheses different approaches were used. To address research questions 1 and 2, data analysis, using SPSS 21, focused on the effect of the independent variable facet availability on dependent variables relating to the usefulness assessment and actual use of the 6 different facets. The data was analysed based on descriptive statistics and by comparing their rank. To test the hypotheses of this research the measures for each of the variables satisfaction, efficiency and effectiveness were examined to determine whether the assumptions for performing parametric or non-parametric variance tests were met. Reviewing the assumptions underlying statistical variance tests, it can be determined that most dependent variables needed to be tested via non-parametric Kruskal-Wallis and Mann-Whitney U tests (Lund Research, 2013a, 2013b). Independent-Samples Mann-Whitney U tests were applied to all independent variables using the two categories of the dependent variable ? whether facets were available in the experimental system or not. Additionally, Kruskal-Wallis tests with sequence of tasks as independent variable to identify potential impacts of when a task was performed were conducted. The data analysis concludes with the main themes that can be found in the qualitative comments provided by participants. 4.2 Results 4.2.1 Actual use and perceived usefulness of facets. In total, the 10 participants provided with the experimental system interacted with facets 327 times. 138 of these interactions employed facets as filters, while the other 189 interactions were conducted to navigate the facet hierarchy. The vast majority of interactions and filter uses, 265 and 119, respectively, were distributed across three facets, Audience, Department, and Type. As outlined in Table 11 participants rated the perceived usefulness of facets almost identically to their actual use when comparing rankings by use and by perceived usefulness. The only difference can be found in rank 5 and 6 in the comparison of total facet use and perceived usefulness. Interestingly, the facet Audience which was found to be of less perceived usefulness in study 1 has the highest ranking for both actual use and perceived usefulness. 54 Table 11 Study 2 - Comparison of Mean Ranks of Perceived Usefulness and Actual Use of Facets Study 2 Study 1 Facet Facet Used As Filter Total Facet Use Perceived Usefulness Perceived Usefulness Use Count Count Rank Use Count Count Rank Mean Mean Rank Mean Rank Audience 51 1 101 1 6.14 1 13 Department 35 2 91 2 6.13 2 1 Type 33 3 73 3 6.00 3 2 Location 15 4 41 4 4.14 4 4 Date Published 4 5 8 6 1.67 5 3 Length 0 6 13 5 1.00 6 9 Facet Used As Filter and Total Facet Use is shown as sum of all scenarios. Ordered by mean rank of perceived usefulness as determined in study 2. N = 10 Table 11: Study 2 - Comparison of Mean Ranks of Perceived Usefulness and Actual Use of Facets A Kruskal-Wallis test was run to determine if there were differences in Facet Use and Facet Filter Use between Types of Search Tasks. Mean Facet Use increased from Doing (Mean = 8.40), Learning (Mean = 11.20), to Known-Item (Mean = 13.10) types of task groups but the differences were not statistically significant, ?2(2) = .598, p = .742. Mean Facet Filter Use increased from Doing (Mean = 2.80), Learning (Mean = 4.70), to Known-Item (Mean = 6.30) types of task groups but the differences were not statistically significant, ?2(2) = .684, p = .710. However, reviewing the use of facets by scenarios based on the total number of interactions, shows that that the use of certain facets varies, as summarized in Table 12. Overall facets were most often used in the Known-Item scenario, while least often used in the Doing scenario. Of the three most prevalently used facets, Audience was most often used in the Known-Item scenario and least often used in the Learning scenario, while the facet Department was used almost as many times in the Learning scenario as in the Known-Item Scenario, and the facet Type was most often used in the Doing scenario. It can also be observed that interactions with the facet Audience in the Known-Item scenario on average resulted in use of the facet as a filter more often, 0.64 times per interaction, than when looking at all interactions across all scenarios with this facet, 0.5 times per interaction. In fact the interactions with facets in the Known-Item scenario all on average resulted in a higher ratio of using the facet as a filter52. 52 The less used facets Date Published, Length, and Location have not been included in this summary as they seem to have been used too few times to establish any meaningful observations when considering the different scenarios. 55 Table 12 Study 2 - Use of Facets by Type of Task Facet Doing Known-Item Learning All Scenarios Facet Used As Filter Total Facet Use Facet Used As Filter Total Facet Use Facet Used As Filter Total Facet Use Facet Used As Filter Total Facet Use Use Count Use Count Ratio Use Count Use Count Ratio Use Count Use Count Ratio Use Count Use Count Ratio Audience 14 33 0.42 28 44 0.64 9 24 0.38 51 101 0.50 Date Published 0 0 - 0 1 0.00 4 7 0.57 4 8 0.50 Department 3 11 0.27 19 42 0.45 13 38 0.34 35 91 0.38 Length 0 5 0.00 0 4 0.00 0 4 0.00 0 13 0.00 Location 0 3 0.00 2 15 0.13 13 23 0.57 15 41 0.37 Type 11 32 0.34 14 25 0.56 8 16 0.50 33 73 0.45 Total 28 84 0.33 63 131 0.48 47 112 0.42 138 327 0.42 Ratio = Facet Used As Filter/Total Facet Use N = 10 Table 12: Study 2 - Use of Facets by Type of Task 4.2.2 Satisfaction. Reviewing the three measures that make up the satisfaction variable, it can be observed that in all cases the baseline system has higher mean scores,as presented in Table 13. The baseline system is indicated to on average have a higher Perceived Ease of Use and provide a higher Level of Satisfaction. On the other hand it seems that participants on average encountered a slightly higher level of challenge when using the baseline system. A Mann-Whitney U test was run for each of the three measures, but no statistically significant difference was found between the baseline and experimental system53. Table 13 Study 2 - Satisfaction ? Comparison of Mean by Facet Availability and Measure Measure Facet Availability No Yes Mean Perceived Ease Of Use 5.13 4.67 Mean Level of Satisfaction 5.20 4.90 Mean Level of Challenge 3.60 3.53 Table 13: Study 2 - Satisfaction ? Comparison of Mean by Facet Availability and Measure 4.2.3 Effectiveness. Reviewing the nine effectiveness measures only very slight differences can be observed, as summarised in Table 14. The baseline system resulted in a higher mean for the number of bookmarked documents directly accessible via the FRED system. It also has a slightly higher mean level of perceived usefulness, perceived knowledge gain, and automatically assessed relevance of bookmarked web pages directly accessible via the FRED systems. A Mann-Whitney 53 More details can be found in Table A44 in the appendix 56 U test was run for each of the three measures, but no statistically significant difference was found between the baseline and experimental system54. Table 14 Study 2 ? Effectiveness: Comparison of Mean by Facet Availability and Measure Measure Facet Availability No Yes Mean Perceived Usefulness 4.70 4.40 Mean Perceived Success 5.00 4.97 Mean Perceived Knowledge Gain 1.30 1.20 Mean Number of Total Documents Bookmarked 4.97 4.97 Mean Number of FRED Accessible Documents Bookmarked 2.47 2.07 Mean Relevance Assessment Manual All URLs 2.03 2.06 Mean Relevance Assessment Participants All URLs 1.50 1.46 Mean Relevance Assessment Manual FRED Accessible URLs 1.80 1.84 Mean Relevance Assessment Participants FRED Accessible URLs 1.45 1.38 Table 14: Study 2 ? Effectiveness: Comparison of Mean by Facet Availability and Measure 4.2.4 Efficiency. Reviewing the 11 efficiency measures that can be used across both systems, multiple differences can be observed as summarised in Table 15. Most notable on the one hand are the lower mean completion time and higher number of results lists viewed when participants used the experimental system. On the other hand participants using the baseline system on average used a higher number of text queries and viewed a higher number of documents. A Mann-Whitney U test was run for each of the 11 measures to determine whether there is a statistically significant difference between the baseline and experimental system. For two of the 11 measures statistically significant differences could be observed55. The Number of Total Documents Viewed Per Results List Viewed was statistically significantly different between the baseline (Mean = 2.20) and experimental system (Mean = 2.11), U = 316, z = -1.982, p = .047. This indicates that there is a statistically significant difference between the total number of results looked at per results list viewed. The Number of FRED Accessible Documents Viewed Per Results List Viewed was statistically significantly different between the baseline (Mean = 0.97) and experimental system (Mean = 0.51), U = 285, z = -2.446, p = .014. This indicates that there is a statistically significant difference between the number of results looked at that are directly accessible from the search system per results list viewed 54 More details can be found in Table A45 in the appendix 55 More details can be found in Table A46 in the appendix 57 Table 15 Study 2 ? Efficiency: Comparison of Mean by Facet Availability and Measure Measure Facet Availability No Yes Mean Completion Time (s) 510.00 461.90 Mean Number of Text Queries 6.03 5.10 Mean Number of Total Documents Viewed 12.13 9.07 Mean Number of FRED Accessible Documents Viewed 5.33 3.60 Mean Number of FRED Results Lists Viewed 8.33 11.30 Mean Number of Total Documents Viewed Per Minute 1.33 1.21 Mean Number of FRED Accessible Documents Viewed Per Minute 0.60 0.57 Mean Number of Total Documents Viewed Per Text Query 2.66 3.35 Mean Number of FRED Accessible Documents Viewed Per Text Query 1.23 1.15 Mean Number of Total Documents Viewed Per Results List Viewed 2.20 2.11 Mean Number of FRED Accessible Documents Viewed Per Results List Viewed 0.97 0.51 Statistically significantly different measures are highlighted in bold. Table 15: Study 2 ? Efficiency: Comparison of Mean by Facet Availability and Measure 4.2.5 Sequence effects. The analysis of sequence effects has been included to determine whether participants behaviour changed over time. The model of the search session states that the earlier time of a session is orientational and the latter is more productive (Marchionini, 1997). This could indicate that the earlier interactions with a search system, for example the first task in a sequence of tasks, are less productive than later interactions, for example the last task in a sequence of tasks. Kruskal-Wallis tests were run for all 27 measures with Sequence of Tasks, 1st, 2nd, or 3rd position of a task in a session, being the independent variable. Pairwise comparisons were performed using Dunn's (1964) procedure with a Bonferroni correction for multiple comparisons. Two of the measures were found to exhibit statistically significant differences56. Looking into more detail, the statistically significant difference can be attributed to the experimental system. In total, across both systems, Perceived Success was statistically significantly different between the different positions in the Sequence of Tasks, ?2(2) = 8.673, p = .013. Perceived Success increased significantly between the 1st (Mean = 4.00) and the 3rd position in the Sequence of Tasks (Mean = 5.70) (p = .013) but not between any other combinations. The Level of Satisfaction was also statistically significantly different between the different positions in the Sequence of Tasks, ?2(2) = 9.379, p = .009. The Level of Satisfaction increased signficantly between the 1st (Mean = 4.05) and the 3rd position in the Sequence of Tasks (Mean = 5.80) (p = .009) but not between any other combinations. 56 More details can be found in Table A47 and A48 in the appendix. 58 Distinguishing between the baseline and experimental system, it can be determined that the difference observed for Perceived Success and Level of Satisfaction only results from differences in use of the experimental system57. No statistically significant difference in the baseline system were observed. Perceived Success was statistically significantly different between the different positions in the Sequence of Tasks for the experimental system, ?2(2) = 8.208, p = .017. Perceived Success increased significantly for the experimental system between the 1st (Mean = 3.60) and the 3rd position in the Sequence of Tasks (Mean = 5.90) (p = .020) but not between any other combinations. The Level of Satisfaction was also statistically significantly different between the different positions in the Sequence of Tasks for the experimental system, ?2(2) = 9.018, p = .011. The Level of Satisfaction increased significantly for the experimental system between the 1st (Mean = 3.40) and the 3rd position in the Sequence of Tasks (Mean = 5.90) (p = .013) but not between any other combinations. 4.2.6 Comments by participants. Reviewing the comments made by participants, it can be observed that the familiarity with the features provided by the baseline system was considered the most useful feature, while several participants also stated that they would like the option to use more sophisticated features. Almost all participants using the baseline system stated that the system being ?[s]imple and straight forward?(P18) and its similarity ?to what we are used to?(P12) was a useful feature. 4 of the 10 participants using the baseline system mentioned that they would like to use features resembling facets, for example ?to specify what kind of document I am looking for\"(P4). 57 More details can be found in Tables A49 to A52 in the appendix. 59 4.3 Summary The findings of this study indicate the following: ? Perceived usefulness and actual use of facets: The facets Audience, Department, and Type are perceived to be most useful and are most often used. ? Satisfaction: No significant differences between the baseline and experimental systems were found. ? Effectiveness: No significant differences between the baseline and experimental system were found. ? Efficiency: o Variance tests indicated that the participants using the experimental system viewed significantly fewer documents per results list viewed, both when considering all documents viewed and only documents directly accessible from the results list. This is not a clear indication of increased efficiency, but does indicate a difference in behaviour patterns when using the two systems. o A trend could be observed in the means of other efficiency measures suggesting that the experimental system was more efficient: on average, participants using the experimental system completed tasks more quickly, issued fewer queries, and viewed fewer documents, although differences were not statistically significant. ? Sequence of tasks: Perceived Success and Level of Satisfaction increased significantly between the first and third task for users of the experimental system, but not for users of the baseline system. ? Overall comparison of baseline and experimental system: Very few differences were found. It seems that, in the framework of this limited study, the added value of facets in the form of greater efficiency was not enough to overcome the benefit of familiarity attributed to the baseline system. 60 5 Discussion 5.1 Overview Studies 1 and 2 were aimed at answering questions about he perceived and actual usefulness of facets depending on types of search tasks, including whether static facets and dynamic task-dependent facets are useful for the discovery of online content. Additionally, study 2 had the purpose to determine whether the hypotheses about higher user satisfaction, effectiveness, and efficiency of systems providing faceted search are valid. To answer the questions, and confirm or refute the hypotheses, this chapter discusses the perceived usefulness and use of facets and compares usability measures. 5.2 Perceived Usefulness and Use of Facets Based on the findings of studies 1 and 2, it can be concluded that there is a set of core facets that is considered most useful and most often used across tasks. Study 1 suggests these core facets to be Department, Type, Date created or published, Geographical area (about), and Timeframe. Study 2 also indicates Department and Type to be amongst these core facets, while it discounts the importance of Date created or published and Geographical area (about). Study 2 also suggests that Audience is an important facet across tasks. Particularly the inconsistency related to the facet Audience is striking, as it was also not considered as highly prevalent when looking at the aggregated result of the systems review. Looking at individual domains, it becomes clearer that the reason why Audience is perceived as more useful and used more is the result of the government content of the FRED system. This could be similar to the finding by Freund (2010), which suggests that another facet - the genre/type of a document - plays an important role in the government domain. Another explanation might be that people are less familiar with the concept of audience, but when given the opportunity to incorporate it in actual searches realize its usefulness. Perceived usefulness and use of facets only seem to be slightly different when looking at the Doing task type. Study 1 concluded that there are statistically significant differences in the perceived usefulness of facets between Doing and Learning task types, which stem from facets Date created or published, Geographic Location (about), and Timeframe. The difference can be attribute to lower mean scores for perceived usefulness in the Doing task type compared to the 61 Learning task type. Study 2 did not find any statistically significant differences between the types of tasks; however, it can be observed that facets were most often used in the Known-Item scenario and were least often used in the Doing scenario. Both studies seem to indicate that the usefulness of facets in Doing scenarios is lower. While systems used to organize knowledge favor learning and finding particular content over performing actions, at least for study 2 an indication of what might be the reason for this can be found. Participants indicated that they had high knowledge about the Doing scenario used in this study. Another reason in case of study 2 might be a result of the set-up of the experimental system. Many of the potentially relevant results were available on the web site of the Canada Mortgage and Housing Corporation, but it was not possible to include them as crawling this web site was not allowed. 5.3 Systems with Facet Capabilities versus Systems without Facet Capabilities There are few clear differences between the system with facet capabilities and the system without facet capabilities. Previous research found that faceted systems led to a higher user satisfaction (Uddin & Janecek, 2007; Yee et al., 2003), but that there might be a cautious reaction initially (Fagan, 2010; Uddin & Janecek 2007; Yee at al. 2003). This might be related to the preference given to simple and familiar interaction styles (Capra et al., 2007), that unfamiliarity often leads users to reject new search interfaces (Yee at al., 2003), and that users tend to prefer simple keyword entry and title listings (English et al., 2002), which can also be observed based on the comments that participants of study 2 made. This preference seems to be based in the mental models individuals form resulting from regular interaction with systems (Chen, Houston, Sewell, and Schatz, 1998). When encountering a new system, a mental model based on previous experience with systems is applied, and in turn, results in a learning curve. Initial interactions with a new system change this mental model. If a system is consistent in behaviour over time, then creating a new or extending an existing mental model is supported which can lead to higher user satisfaction. Looking at the measures recorded in study 2 over time, it seems that the impact of growing familiarity and the formation of a mental model can be observed on both user satisfaction and effectiveness. Level of Satisfaction and Perceived Success were indicated to be statistically significantly different between the different positions in the sequence of tasks. For both measures this statistically significant difference stems from the contrast between the 1st and 3rd position in 62 the Sequence of Tasks in the use of the experimental system. The increase in Level of Satisfaction and Perceived Success for the experimental system is markedly higher than for the baseline system. One explanation for this phenomenon could be that the experimental system provides higher effectiveness and satisfaction over time. Another possible explanation might be result of a decreasing number of facet interactions over time. Out of the 327 facet interactions in total 177 occurred while performing task 1 and decreased down to 58 for task 3. While this could support that participants gain a higher knowledge about the facet hierarchies and require less interactions to find what they are looking for, which indicates a higher efficiency, it could also be evidence of fatigue. During the first task participants might be very enthusiastic and consequently trying to make the most use of the facet feature while they just want to finish the later tasks. It might also make sense to not include the results of the first task in experimental user studies in the analysis, or include an orientational \"warm-up\" for users as done by English et al. (2002). Comparing measures of efficiency between the baseline and experimental system, no clear determination which system provides a higher efficiency can be found, however statistically significant differences can be found that suggest that that there is more emphasis on obtaining results list in the experimental system, and more emphasis on assessing the documents retrieved in the baseline system. It can also be observed that participants using the experimental system completed tasks more quickly. Vaughan and Dillon (2005) found statistically significant differences in the completion speed depending on whether a search system conforms to or at least approximates the structure and mental representation of a domain. In their study, they observed that participants presented with a systems that conforms or approximates the mental representation of a domain complete tasks more quickly. In essence, this is again related to the mental model that participants have established or are forming regarding a particular domain or system. There seems to be the potential in faceted systems to represent a domain more clearly and in turn this could support the formation of a mental model. In this context, it is also interesting to note that on average participants using the faceted system took a little bit longer to complete their first task (551 seconds versus 536 seconds), while completing tasks much more quickly, by more than a minute, when performing their second and third tasks. This seems to be similar to a finding by Kules et al. (2009), which suggests that participants not familiar with a topic spent more time to look at facets to determine how to proceed with their search. The two search outcomes provided at the same time - a list of results and a navigation structure - seem to 63 support users in grasping the information space (Chen et al., 1998), and thus in more easily forming a more comprehensive mental model about the domain and system. Only looking at the activities not related to facets, participants using the experimental system performed fewer interactions with the interface on average, with the exception of the number of results lists viewed. As the number of documents/web pages viewed is higher in the baseline system, this result seems to not match the outcome expected by Pratt et al. (1999) in that the number of results viewed is expected to be higher in faceted systems, unless they considered the number of results lists viewed as well, which is not entirely clear from their elaborations. However, when taking into account activities related to facets, then on average participants using the experimental system performed 45 interface interactions per task, while participants not having the possibility to use facets only performed 31 interactions per task. Hence, it seems that while interactions with non-facet interface elements decreased with the availability of facets, the interactions with facet interface elements increased to a higher degree. This finding seems to be contradictory to Suthcliffe, Ennis, and Watkinson?s (2000) finding, which being over a decade old might be outdated by now, that users usually do not use more advanced features of an interface and stick to simple text queries. Some participants in study 2, only having been provided with a simple text query feature and a results list as interface elements, mentioned that advanced features, such as filters or Boolean operators would be useful. It might be that more common use of advanced features and a better integration into user interfaces over the last decade has led to users wanting more flexibility in how to search for information. For example, a faceted approach can be easily used to create queries containing Boolean operators which support non-expert users in particular (English et al., 2002). Another reason for participants of study 2 using facets as an example of advanced features, or wanting to use more advanced features, might be a result of their relatively high self-reported skill level in searching the Internet. 5.4 Summary Research question 1. Considering types of search tasks, is there a difference in the perceived usefulness of facets for discovery of online content? Does the actual use of facets in search systems vary by type of task? 64 According to the findings of study 1, facets perceived to be most useful are the same across the three investigated types of search tasks. There are slight differences in their order. Although there are some statistically significant differences, it still can be suggested that there does not seem to be a very strong difference in the perceived usefulness of facets by task. Similarly, concerning the actual use of facets, based on findings of study 2, the same three facets were used most often in all three types of tasks. However, task-based differences can be found in the number of interactions in total and per facet, as well as in the ratio at which facet interactions resulted in actually using facets as filters. Research question 2. Are static facets and dynamic task-dependent facets useful for discovery of online content? As the same facets seem to be preferred for all investigated tasks, there are no indications that dynamic task-dependent facets are a necessary feature for search systems. The core facets identified in study 1 and in study 2 seem to be useful across different types of search tasks, but may possibly be somewhat different, depending on the content?s domain. Hence, it can be suggested that these facets used as static facets - i.e. static in terms of being available in the interface, and showing facet labels dynamically depending on search queries and the search systems index - are useful. Hypothesis 1. Search systems providing faceted search lead to a higher user satisfaction compared to search systems without this capability. Considering all three tasks participants of study 2 performed, this hypothesis would need to be rejected, as there are no significant differences in satisfaction across systems. Taking into account changes in perception over time, it seems as though that user satisfaction grew and would perhaps lead to significant differences. Long-term studies are needed to provide more conclusive results. Hypothesis 2. Search systems providing faceted search lead to a higher effectiveness compared to search systems without this capability. This hypothesis is rejected as no significant differences were found on measures of effectiveness. Over time, participants seemed to perceive a higher success when using the experimental system. 65 To determine whether this has a significant impact, long-term studies are needed to provide more conclusive results. Hypothesis 3. Search systems providing faceted search lead to a higher efficiency compared to search systems without this capability. Although significant differences were found in two measures, this hypothesis is rejected as they cannot provide a conclusive result on their own. 66 6 Conclusion 6.1 Summary This research investigated the perceived usefulness and actual use of facets in the discovery of online content in the context of Doing, Known-Item, and Learning tasks. Study 1 of this research, a system review and a questionnaire-based online survey, indicates that the usefulness of facets is perceived quite similarly across tasks. A statistically significant difference has been found in the perceived usefulness between the Doing and Learning tasks, with the Doing tasks showing lower perceived usefulness scores. Still, the findings of study 1 do not support the idea of a task-dependent adaption of facet sets in search user interfaces. Study 2 of this research, a between-subjects experimental user study with a baseline system without facets and an experimental system with facets, compared the perceived usefulness of study 1 to actual use of facets. The study determined that there can be differences in the actual use and perception of usefulness, in this case for the facet Audience which was not perceived as highly useful in study 1, while it was used most often in the user experiments conducted in study 2. One explanation could be that study 2 was focused on government content while study one incorporated government, library, and commercial systems. Looking at the systems review conducted in study 1, it can be observed that Audience is more prevalently employed in search systems in the government domain. Study 2 also investigated whether a system providing faceted search leads to a higher user satisfaction, effectiveness, and efficiency compared to a system not providing this capability. It seems that the added value of facets could not overcome the familiarity attributed to the baseline system. No statistically significant differences between the two systems were found in terms of user satisfaction and effectiveness. While statistically significant differences were found in 2 of 11 measures used to determine efficiency, these do not provide a basis for a conclusive evaluation. Considering participants? interactions with the systems over time, the study found that there are statistically significant differences in Perceived Success, a measure of effectiveness, and Level of Satisfaction, a measure of satisfaction, between the first and third tasks performed by participants, with the third task showing higher scores. These differences have their origin in statistically significant differences between the first and third tasks in the 67 experimental system. Hence, it seems that satisfaction and effectiveness in terms of the experimental system grew over time and would perhaps lead to significant differences when conducting long-term studies. 6.2 Limitations When considering the findings of this research, several limitations have to be noted and taken into account. For both studies, the scenarios used were imposed on the participants; hence, the scenarios might not have constituted something interesting and relevant to all participants (Borlund, 2003). And while at least for study 2, an indication can be made that the scenarios were perceived to be relatively high in realism on a 7-point Likert scale, this limitation should still be kept in mind as natural search tasks originating from participants themselves should be seen as more valid. The small number of scenarios in study 2, one representing each task type, which was chosen in conjunction with a manageable number of only 1,005 indexed web pages, is not a sufficient basis for generalization of the findings of this study beyond these examples. Having only used this small number of scenarios and web pages in conjunction with only 6 facets, there might be a bias towards one or the other facet based on the scenario. For example, adding or leaving out location aspects or date aspects in a scenario description most likely impacted the perceived usefulness and use of related facets. The arrangement of the facet hierarchy, created via card sorting and group labelling by a two-person focus group, might also have affected participants? perceptions about the usefulness of facets, as they might have expected a different structure or labels. This expectation relates to the impact of exposure to a system over time outlined by Vaughan and Dillon (2005). While three different tasks were performed by participants, there is a clear indication that perceptions and use of systems changed over time, particularly related to the system providing facets. It could be that making observations over an even longer time might result in additional changes in user perception and behaviour. Beyond these limitations, the skew towards participants in younger age groups being students, having a high level of education, and a high self-rated skill in searching the Internet does not make the findings of this research applicable to the general public. 6.3 Future research There are several main approaches to extend this research. First, particularly to address the limitations stated in terms of system set-up, it is necessary to expand the content indexed in the 68 experimental system. For this, it is necessary to employ more automated approaches for classifying web pages in terms of different facets, as already done with the Location facet. In this context, it seems prudent to review the algorithm used by Pratt et al. (1999) for dynamic categorization of results into a hierarchical organization. For the Type facet, a proof of concept has already been created, but was not incorporated in this study and needs to be extended to other facets. In addition, the use of eye gaze tracking, similar to the approach used by Kules et al. (2009), is an additional approach to measure user interaction with search systems, which was considered, but not used in this research. For taking into account potential changes in perception or behaviour over time, it might be prudent to not only conduct a study with one experimental session, but with two sessions being separated by a few days. And lastly, it should be considered to set up the experimental user study in a way that non-local participants can be recruited by making the user study available online. This could result in increasing the diversity of participants. Research beyond this study should investigate whether the idea of dynamic task-dependent adaptation of facet sets might be applicable in the context of a single domain only instead of across domains. It would also be interesting to investigate the differences in results in terms of the impact of facets on user behaviour between short-term and long-term studies. 6.4 Implications As already outlined by Ingwersen and J?rvelin (2005), designing and evaluating information retrieval systems is multi-dimensional and complex. Focus needs to be on supporting a system?s user by providing means to make information seeking ?faster, with less resources, [and] with better quality? (p. 314), or in other words, providing a more efficient, effective, and satisfactory experience. Additionally, there is no one ultimate combination of these three dimensions of usability to strive for. Stressing one or the other dimension seems to be highly dependent on the context. The type of search task an individual is trying to accomplish is part of this broader context. Bearing in mind the limitation of this research, it can be suggested that the facets Audience, Date created or published, Department (Organization), Geographical area (about), Timeframe (Coverage About), and Type of document, or some kind of derivative thereof should be considered for inclusion in faceted search systems in the government domain. The effect of changes of perception about and behaviour when using search systems should also be considered in design and evaluation, both in professional as well as research projects. 69 References Adkisson, H. P. (2005). Web Design Practices | Faceted Classification. Retrieved January 21, 2013, from http://www.webdesignpractices.com/navigation/facets.html Baeza-Yates, R., & Ribeiro-Neto, B. (2011). Modern information retrieval: the concepts and technology behind search (Second edition.). New York: Addison Wesley. Ben-Yitzhak, O., Golbandi, N., Har?El, N., Lempel, R., Neumann, A., Ofek-Koifman, S., ? Yogev, S. (2008). Beyond basic faceted search. In Proceedings of the international conference on Web search and web data mining (pp. 33?44). New York, NY, USA: ACM. doi:10.1145/1341531.1341539 Borlund, P. (2000). Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation, 56(1), 71?90. Borlund, P. (2003). The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3). Retrieved from http://informationr.net/ir/8-3/paper152.html Borlund, P., & Schneider, J. W. (2010). Reconsideration of the simulated work task situation: A context instrument for evaluation of information retrieval interaction. In Proceedings of the third symposium on Information interaction in context (pp. 155?164). Retrieved from http://dl.acm.org/citation.cfm?id=1840808 Brinck, T., Gergle, D., & Wood, S. D. (2002). Designing Web sites that work usability for the Web. San Francisco: Morgan Kaufmann Publishers. Retrieved from 70 http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=81858 Broughton, V. (2006). The need for a faceted classification as the basis of all methods of information retrieval. Aslib Proceedings, 58(1/2), 49?72. doi:10.1108/00012530610648671 Buie, E., & Murray, D. (2012). Usability in government systems user experience design for citizens and public servants. Burlington: Elsevier Science. Bystr?m, K., & Hansen, P. (2005). Conceptual framework for tasks in information studies. Journal of the American Society for Information Science and Technology, 56(10), 1050?1061. Bystr?m, Katriina, & Hansen, P. (2002). Work tasks as units for analysis in information seeking and retrieval studies (pp. 239?252). Greenhood, United States: Libraries Unlimited. Bystr?m, Katriina, & J?rvelin, K. (1995). Task complexity affects information seeking and use. Information Processing & Management, 31(2), 191?213. doi:10.1016/0306-4573(95)80035-R Cabinet Office. (2006). e-Government Metadata Standard V3.1 (Standard). Cabinet Office. Retrieved from http://webarchive.nationalarchives.gov.uk/+/http://www.cabinetoffice.gov.uk/media/273711/egmsv3-1.pdf Capra, R., Marchionini, G., Oh, J. S., Stutzman, F., & Zhang, Y. (2007). Effects of structure and interaction style on distinct search tasks. In Proceedings of the 7th ACM/IEEE-CS joint 71 conference on Digital libraries (pp. 442?451). New York, NY, USA: ACM. doi:10.1145/1255175.1255267 Cellan-Jones, R. (2012, July 12). Facebook ?likes? and adverts? value doubted. BBC News - Technology. News. Retrieved June 21, 2013, from http://www.bbc.co.uk/news/technology-18813237 Chen, H., Houston, A. L., Sewell, R. R., & Schatz, B. R. (1998). Internet browsing and searching: User evaluations of category map and concept space techniques. Journal of the American Society for Information Science, 49(7), 582?603. doi:10.1002/(SICI)1097-4571(19980515)49:7<582::AID-ASI2>3.0.CO;2-X Church, A. H. (1993). Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis. The Public Opinion Quarterly, 57(1), 62?79. doi:10.2307/2749438 Cobanoglu, N., & Cobanoglu, C. (2003). The effect of incentives in web surveys: application and ethical considerations. International Journal of Market Research, 45(4), 475?488. Cool, C., & Belkin, N. J. (2011). Interactive information retrieval: history and background. In I. Ruthven & D. Kelly (Eds.), Interactive Information Seeking, Behaviour and Retrieval. London, U.K. Dakka, W., Dayal, R., & Ipeirotis, P. (2006). Automatic discovery of useful facet terms. In SIGIR Faceted Search Workshop. Retrieved from http://www.ipeirotis.com/wp-content/uploads/2012/01/sigir_faceted2006.pdf Dakka, W., Ipeirotis, P. G., & Wood, K. R. (2005). Automatic construction of multifaceted browsing interfaces. In Proceedings of the 14th ACM international conference on 72 Information and knowledge management (pp. 768?775). Retrieved from http://dl.acm.org/citation.cfm?id=1099738 Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys?: the tailored design method. Hoboken, N.J.: Wiley & Sons. Donn, K., Plaisant, C., & Shneiderman, B. (1996). Query previews in networked information systems. In Research and Technology Advances in Digital Libraries, 1996. ADL?96., Proceedings of the Third Forum on (pp. 120?129). Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=502522 Druckman, J. N., & Kam, C. D. (2009). Students as Experimental Participants: A Defense of the ?Narrow Data Base? (SSRN Scholarly Paper No. ID 1498843). Rochester, NY: Social Science Research Network. Retrieved from http://papers.ssrn.com/abstract=1498843 Dunn, O. J. (1964). Basic statistics: a primer for the biomedical sciences. New York: Wiley. English, J., Hearst, M., Sinha, R., Swearingen, K., & Yee, K.-P. (2002). Flexible Search and Navigation Using Faceted Metadata. University of Berkeley. Fagan, J. C. (2010). Usability Studies of Faceted Browsing: A Literature Review. Information Technology and Libraries, 29(2), 58?66. doi:10.6017/ital.v29i2.3144 Fountain, J. E. (2001). Building the Virtual State: Information Technology and Institutional Change. Brookings Institution Press. Freund, L. (2008a). Exploiting Task-document Relations in Support of Information Retrieval in the Workplace. Retrieved from https://tspace.library.utoronto.ca/handle/1807/16762 73 Freund, L. (2008b). Putting Digital Government in Context (pp. 75?79). Presented at the Second International Workshop on Adaptive Information Retrieval, London, U.K. Retrieved from http://www.dcs.gla.ac.uk/workshops/air2008/proceedings.pdf Freund, L. (2011). Document Usefulness: taking Genre to Task. Presented at the Workshop on Human-Computer Interaction and Information Retrieval, Mountain View, CA. Freund, L., & Berzowska, J. (2010). The Goldilocks Effect: Task-Centred Assessments of E-Government Information. Presented at the Annual Conference of the American Society for Information Science and Technology, Pittsburgh, PA. Retrieved from http://www.asis.org/asist2010/proceedings/proceedings/ASIST_AM10/submissions/261_Final_Submission.pdf Freund, L., Berzowska, J., & Hopton, L. (2011). E-informing the public: communicative intents in the production of online government information. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries (pp. 443?444). New York, NY, USA: ACM. doi:10.1145/1998076.1998176 Freund, L., Jinglewski, M., & Kessler, K. (2012). Introducing FRED: Faceted Retrieval of E-government Documents. Presented at the Annual Conference of the American Society for Information Science and Technology, Baltimore, MD. Freund, L., Kopak, R., Kessler, K., & Vantyghem, T. (2013). Designing for E-Government Search. In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval. Vancouver, BC, Canada. 74 Freund, L., & Nilsen, C. (2008). Assessing a genre-based approach to online government information. Presented at the Annual Conference of the Canadian Association for Information Science, Vancouver, BC, Canada. Retrieved from http://www.cais-acsi.ca/proceedings/2008/freund_2008.pdf Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987). The vocabulary problem in human-system communication. Communications of the ACM, 30(11), 964?971. Gerben, C. (2010, September 20). Why I (just may) love Facebook Ads for Collecting Data. Retrieved from http://hastac.org/blogs/cgerben/why-i-just-may-love-facebook-ads-collecting-data Gruber, T. R. (1993). A translation approach to portable ontology specifications. Knowledge acquisition, 5(2), 199?220. Hansen, P. (1999). User Interface Design for IR Interaction. A Task-Oriented Approach. In T. Saracevic, P. Ingwersen, & P. Vakkari (Eds.), CoLIS 3. Proceedings of the Third International Conference on the Conceptions of the Library and Information Science (pp. 191?205). Dubrovnik, Croatia. Hart, C. (1998). Doing a literature review: releasing the social science research imagination. London: Sage Publications. Hearst, M. A. (2008). UIs for Faceted Navigation Recent Advances and Remaining Open Problems. In Proceedings of the Second Workshop on Human-Computer Interaction and Information Retrieval. 75 Hearst, M. A. (2011). User interfaces for search. In Ricardo Baeza-Yates & B. Ribeiro-Neto (Eds.), Modern information retrieval?: the concepts and technology behind search (Second edition., pp. 21?54). New York: Addison Wesley. Hearst, M., Elliott, A., English, J., Sinha, R., Swearingen, K., & Yee, K.-P. (2002). Finding the flow in web site search. Commun. ACM, 45(9), 42?49. doi:10.1145/567498.567525 Holiday, R. (2012, May 17). Why I Lost My Faith In Facebook Advertising. Why I Lost My Faith In Facebook Advertising - Forbes. News. Retrieved June 21, 2013, from http://www.forbes.com/sites/ryanholiday/2012/05/17/why-i-lost-my-faith-in-facebook-advertising/ Ingwersen, P., & J?rvelin, K. (2005). The Turn: Integration of Information Seeking and Retrieval in Context. Springer. James, J. M., & Bolstein, R. (1990). The Effect of Monetary Incentives and Follow-Up Mailings on the Response Rate and Response Quality in Mail Surveys. The Public Opinion Quarterly, 54(3), 346?361. doi:10.2307/2749372 James, J. M., & Bolstein, R. (1992). Large Monetary Incentives and Their Effect on Mail Survey Response Rates. Public Opinion Quarterly, 56(4), 442?453. doi:10.1086/269336 J?rvelin, K., & Ingwersen, P. (2004). Information seeking research needs extension towards tasks and technology. Information Research, 10(1), 10?1. Johnson, T. R., & McLaughlin, S. D. (1990). The GMAT Registrant Survey Research Design Report. Graduate Management Admission. 76 Kim, J. (2012). Scenarios in information seeking and information retrieval research: A methodological application and discussion. Library & Information Science Research, 34(4), 300?307. doi:10.1016/j.lisr.2012.04.002 Kules, B., & Capra, R. (2012). Influence of training and stage of search on gaze behavior in a library catalog faceted search interface. Journal of the American Society for Information Science and Technology, 63(1), 114?138. doi:10.1002/asi.21647 Kules, B., Capra, R., Banta, M., & Sierra, T. (2009). What do exploratory searchers look at in a faceted search interface? In Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries (pp. 313?322). New York, NY, USA: ACM. doi:10.1145/1555400.1555452 Kules, B., & Shneiderman, B. (2008). Users can change their web search tactics: Design guidelines for categorized overviews. Inf. Process. Manage., 44(2), 463?484. doi:10.1016/j.ipm.2007.07.014 Li, Y. (2008). Exploring the relationships between work task and search task in information search. Journal of the American Society for information Science and Technology, 60(2), 275?291. Li, Y., & Belkin, N. J. (2008). A faceted approach to conceptualizing tasks in information seeking. Inf. Process. Manage., 44(6), 1822?1837. doi:10.1016/j.ipm.2008.07.005 Liu, D., Lease, M., Kuipers, R., & Bias, R. (2012). Crowdsourcing for Usability Testing. arXiv:1203.1468. Retrieved from http://arxiv.org/abs/1203.1468 77 Lund Research. (2013a). Kruskal-Wallis H Test using SPSS. Retrieved June 24, 2013, from https://statistics.laerd.com/spss-tutorials/kruskal-wallis-h-test-using-spss-statistics.php Lund Research. (2013b). One-way ANOVA in SPSS. Retrieved June 24, 2013, from https://statistics.laerd.com/spss-tutorials/one-way-anova-using-spss-statistics.php Marchionini, G. (1989). Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science, 40(1), 54?66. doi:10.1002/(SICI)1097-4571(198901)40:1<54::AID-ASI6>3.0.CO;2-R Marchionini, G. (1997). Information Seeking in Electronic Environments. Cambridge, MA: Cambridge University Press. Marchionini, G. (2006). Exploratory search: from finding to understanding. Commun. ACM, 49(4), 41?46. doi:10.1145/1121949.1121979 Marchionini, G., & Levi, M. (2003). Digital government information services: the Bureau of Labor statistics case. interactions, 10(4), 18?27. doi:10.1145/838830.838864 Mills, J. (2004). Faceted Classification and Logical Division in Information Retrieval. Retrieved from http://hdl.handle.net/2142/1687 Morton, R., & Williams, K. (2010). Experimentation in Political Science. In J. Box-Steffensmeier (Ed.), The Oxford Handbook of Political Methodology (pp. 339?356). New York, NY, USA: Oxford University Press. Morville, P. (2007). Information architecture for the World Wide Web (3rd ed.). Sebastopol, CA: O?Reilly. 78 National Archives of Australia. (2010). AGLS Metadata Standard - Part 1 ? Reference Description (Standard No. AS 5044-2010). Canberra: National Archives of Australia. Retrieved from http://www.agls.gov.au/pdf/AGLS%20Metadata%20Standard%20Part%201%20Reference%20Description.PDF Nielsen, J. (2006, June 26). Quantitative Studies: How Many Users to Test? Retrieved June 30, 2013, from http://www.nngroup.com/articles/quantitative-studies-how-many-users/ Orlikowski, W. J., & Yates, J. A. (1994). Genre repertoire: The structuring of communicative practices in organizations. Administrative science quarterly, 541?574. Porter, S. R., & Whitcomb, M. E. (2003). The Impact of Lottery Incentives on Student Survey Response Rates. Research in Higher Education, 44(4), 389?407. doi:10.2307/40197312 Pratt, W., Hearst, M. A., & Fagan, L. M. (1999). A knowledge-based approach to organizing retrieved documents. In Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence (pp. 80?85). Menlo Park, CA, USA: American Association for Artificial Intelligence. Retrieved from http://dl.acm.org/citation.cfm?id=315149.315232 Ranganathan, S. R. (1950). Colon classification. Madras Library Association. Ranganathan, S. R. (1951). Philosophy of library classification. E. Munksgaard. 79 Ranganathan, S. R. (1962). Elements of library classification: based on lectures delivered at the University of Bombay in December 1944 and in the schools of librarianship in Great Britain in December 1956. Asia Pub. House. Ranganathan, S. R. (1963). Colon classification: basic classification. Asia Pub. House. Samuels, D. J., & Zucco, C. (2012). Using Facebook as a Subject Recruitment Tool for Survey-Experimental Research. SSRN eLibrary. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2101458 Saracevic, T. (2011). Foreword. In I. Ruthven & D. Kelly (Eds.), Interactive Information Seeking, Behaviour and Retrieval. London, U.K. Savolainen, R. (1995). Everyday life information seeking: Approaching information seeking in the context of ?way of life.? Library & Information Science Research, 17(3), 259?294. Shneiderman, B. (1994). Dynamic queries for visual information seeking. Software, IEEE, 11(6), 70?77. Shneiderman, B. (1998). Designing the user interface: strategies for effective human-computer-interaction. Reading, Mass: Addison Wesley Longman. Simon, H. (1971). Designing Organizations for an Information-Rich World. In M. Greenberger (Ed.), Computers, communications, and the public interest (pp. 37?72). Simon, H. A., Egidi, M., & Marris, R. (2008). Economics, bounded rationality, and the cognitive revolution. Cheltenham, UK?; Northampton, MA: Edward Elgar. 80 Spiteri, L. F. (1998a). A simplified model for facet analysis. Canadian journal of Information and Library Science, 23(1-2), 1?30. Spiteri, L. F. (1998b). The Use of Facet Analysis in Information Retrieval Thesauri: An Examination of Selected Guidelines for Thesaurus Construction. Cataloging & Classification Quarterly, 25(1), 21?37. doi:10.1300/J104v25n01_03 Stoica, E., Hearst, M. A., & Richardson, M. (2007). Automating creation of hierarchical faceted metadata structures. In Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2007) (pp. 244?251). Retrieved from http://acl.ldc.upenn.edu/N/N07/N07-1031.pdf Suthcliffe, M., Ennis, M., & Watkinson, S. J. (2000). Empirical studies of end-user information searching. Journal of the American Society for Information Science, 51(13), 1211?1231. Taylor, R. (1991). Information use environments. In B. Dervin & M. J. Voigt (Eds.), Progress in communication sciences (pp. 217?255). Ablex Publishing. Toms, E. G. (2011). Task-based information searching and retrieval. In I. Ruthven & D. Kelly (Eds.), Interactive Information Seeking, Behaviour and Retrieval. London, U.K. Toms, E. G., O?Brien, H., Mackenzie, T., Jordan, C., Freund, L., Toze, S., ? MacNutt, A. (2008). Task Effects on Interactive Search: The Query Factor. In N. Fuhr, J. Kamps, M. Lalmas, & A. Trotman (Eds.), Focused Access to XML Documents (pp. 359?372). Springer Berlin Heidelberg. Retrieved from http://link.springer.com/chapter/10.1007/978-3-540-85902-4_31 81 Treasury Board of Canada Secretariat. (2012, March 21). Metadata. Retrieved February 5, 2013, from http://www.tbs-sct.gc.ca/im-gi/imrc-crgi/metadata-metadonnees-eng.asp#s4 Tunkelang, D. (2006). Dynamic Category Sets: An Approach for Faceted Search. Tunkelang, D. (2009). Faceted Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1), 1?80. doi:10.2200/S00190ED1V01Y200904ICR005 Uddin, M. N., & Janecek, P. (2007). Performance and usability testing of multidimensional taxonomy in web site search and navigation. Performance Measurement and Metrics, 8(1), 18?33. doi:10.1108/14678040710748058 Vakkari, P. (1999). Task complexity, problem structure and information actions: Integrating studies on information seeking and retrieval. Information Processing & Management, 35(6), 819?837. doi:10.1016/S0306-4573(99)00028-X Vakkari, P. (2003). Task-based information searching. Annual Review of Information Science and Technology, 37(1), 413?464. doi:10.1002/aris.1440370110 Vaughan, M. W., & Dillon, A. (2006). Why structure and genre matter for users of digital information: A longitudinal experiment with readers of a web-based newspaper. International Journal of Human-Computer Studies, 64(6), 502?526. doi:10.1016/j.ijhcs.2005.11.002 Vicente, K. J. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Lawrence Erlbaum. Walker, G., Janes, J., & Tenopir, C. (1999). Online retrieval?: a dialogue of theory and practice. Englewood, Colo.: Libraries Unlimited. 82 White, R. W., Kules, B., Drucker, S. M., & schraefel, mc. (2006). Supporting exploratory search - Introduction. Commun. ACM, 49(4), 36?39. doi:10.1145/1121949.1121978 White, R. W., & Roth, R. A. (2009). Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1), 1?98. Wildemuth, B. M., & Freund, L. (2012). Assigning search tasks designed to elicit exploratory search behaviors. In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval (pp. 4:1?4:10). Boston, MA, USA: ACM. doi:10.1145/2391224.2391228 Worstall, T. (2012, January 10). Facebook: Ad Clicks Show We?re Bad Value So Let?s Measure Advertising On Facebook Another Way. Facebook: Ad Clicks Show We?re Bad Value So Let?s Measure Advertising On Facebook Another Way - Forbes. News. Retrieved June 21, 2013, from http://www.forbes.com/sites/timworstall/2012/10/01/facebook-ad-clicks-show-were-bad-value-so-lets-measure-advertising-on-facebook-another-way/ Yee, K., Swearingen, K., Li, K., & Hearst, M. (2003). Faceted metadata for image search and browsing (pp. 401?408). Presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, ACM. doi:10.1145/642611.642681 83 Appendix A ? Summaries of Assessment of Use of Facets and Filters Table A1 Systems Review - List of Departments and Search Systems Reviewed ? Government of Australia Department/Search System Link Department of Agriculture, Fisheries and Forestry http://agencysearch.australia.gov.au/search/search.cgi?query=&collection=agencies&form=simple&profile=daff Department of Climate Change and Energy Efficiency http://www.climatechange.gov.au/search.aspx?query=test&collection=agencies&profile=climatechange Department of Defence http://search.defence.gov.au/search?site=default_collection&client=default_frontend&output=xml_no_dtd&proxystylesheet=default_frontend&q=&sa=Search&ie=UTF-8&ip=128.189.137.251&access=p&sort=date:D:L:d1&entqr=0&entqrm=0&oe=UTF-8&ud=1&proxycustom=%3CADVANCED/%3E Department of Education, Employment and Workplace Relations http://deewr.gov.au/search/site Department of Families, Housing, Community Services and Indigenous Affairs http://agencysearch.australia.gov.au/search/search.cgi?profile=fahcsia_preview&collection=agencies&query=&form=simple Department of Finance and Deregulation http://www.finance.gov.au/search/advanced_search.html Department of Foreign Affairs and Trade http://agencysearch.australia.gov.au/search/search.cgi?query=&collection=agencies&profile=dfat&form=simple Department of Health and Ageing http://www.health.gov.au//internet/main/publishing.nsf/Content/Home Department of Human Services http://agencysearch.australia.gov.au/search/search.cgi?query=&form=custom&profile=humanservicesportfolio&collection=agencies&scope=%2F Department of Industry, Innovation, Science, Research and Tertiary Education http://www.innovation.gov.au/Search/Pages/advanced.aspx Department of Infrastructure and Transport http://search.infrastructure.gov.au/search/search.cgi?collection=Infrastructure&form=advanced Department of Regional Australia, Regional Development and Local Government http://search.regional.gov.au/search/search.cgi?collection=regional&form=simple_regional&query=&Submit=Go Department of Resources, Energy and Tourism http://www.ret.gov.au/Pages/default.aspx Department of Sustainability, Environment, Water, Population and Communities http://agencysearch.australia.gov.au/search/search.cgi?collection=agencies&profile=environment&form=advanced Department of the Prime Minister and Cabinet http://agencysearch.australia.gov.au/search/search.cgi?query=&collection=agencies&form=simple&profile=pmc Department of Veteran?s Affairs http://www.dva.gov.au/DVASearchResults.aspx?k= Parliament of Australia search system http://parlinfo.aph.gov.au/parlInfo/search/search.w3p;adv=yes Search system across all government content http://australia.gov.au/search?collection=gov_all&coverage=all&gscope1=&form=simple&sort=&num_ranks=3&extra_all_num_ranks=3&searchAgain=false&advancedSearch=false&query=&location=&query_and=&query_phrase=&query_not=&scope=&meta_f_sand=# The Treasury (Department) http://agencysearch.australia.gov.au/search/search.cgi?collection=agencies&profile=treasury&query=&scope_disable=off Note: Number of departments n = 18; Last accessed April 22nd, 2013 Table A1: Systems Review - List of Departments and Search Systems Reviewed ? Government of Australia 84 Table A2 Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Australia Facet/Filter Category Form of Use Score Audience Metadata standard; as facet by Department of Education, Employment and Workplace Relations; as filter by Department of Industry, Innovation, Science, Research and Tertiary Education, Department of Human Services 2 Availability Metadata standard 1 Contributor Metadata standard 1 Coverage - Jurisdiction Metadata standard 1 Coverage - Spatial Metadata standard; rudimentary in search system across all government content 1 Creator Metadata standard 1 Date - Availability Metadata standard 1 Date - Copyright Metadata standard 1 Date - extracted Metadata standard 1 Date - issued Metadata standard 1 Date - licensed Metadata standard 1 Date - modified Metadata standard; as facet by Parliament of Australia search system 1 Date - published (Date ? created) Metadata standard 1 Date ? Validity Metadata standard 1 Function Metadata standard 1 Item ? Extent (Format ? extent) Metadata standard 1 Item ? Format (File Format) Metadata standard; search system across all government content; as filter by Department of Defence 3 Language Metadata standard 1 Mandate Metadata standard 1 Organization (Type ? Category) Metadata standard; Rudimentary in search system across all government content; as facet sub department by Department of Infrastructure and Transport; as filter by Department of Agriculture, Fisheries and Forestry, Department of Defence, Department of Finance and Deregulation, Department of Health and Ageing, Department of Families, Housing, Community Services and Indigenous Affairs, Department of Regional Australia, Regional Development and Local Government 3 Publisher Metadata standard 1 Relation Metadata standard 1 Rights - Access Rights (License) Metadata standard 1 Rights - Rights Holder Metadata standard 1 Source Metadata standard 1 Status Metadata standard 1 Subject (Topic) Metadata standard; as filter by Department of Industry, Innovation, Science, Research and Tertiary Education, Department of Infrastructure and Transport, Department of Agriculture, Fisheries and Forestry, Department of Finance and Deregulation, Department of Health and Ageing 2 Time Frame (Coverage ? Temporal) Metadata standard; as filter by Department of Infrastructure and Transport 1 Type - Aggregation Level Metadata standard 1 Type - Document Metadata standard; rudimentary in search system across all government content; as facet by Parliament of Australia search system; as filter by Department of Industry, Innovation, Science, Research and Tertiary Education, Department of Agriculture, Fisheries and Forestry, Department of Finance and Deregulation 2 Type ? Service Metadata standard 1 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. No facets or filter categories were found in the search systems of these departments: Department of Climate Change and Energy Efficiency, Department of Foreign Affairs and Trade, Department of Resources, Energy and Tourism, Department of Sustainability, Environment, Water, Population and Communities, Department of the Prime Minister and Cabinet, Department of Veteran?s Affairs, The Treasury (Department) Note: Number of departments n = 18; Status as of April 22nd, 2013 Table A2: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Australia 85 Table A3 Systems Review - List of Departments and Search Systems Reviewed ? Government of Canada Department/Search System Link Aboriginal Affairs and Northern Development http://srch.aadnc-aandc.gc.ca/index.html?ql=a&charset=iso-8859-1&qp=url%3A*-eng.* Agriculture and Agri-Food Canada http://srch-rech.agr.gc.ca/srch-rech/aafc-aac/search-recherche.jsp?advanced=true&FileFormatBox=html&lang=eng Canada Revenue Agency http://www.cra-arc.gc.ca/ebci/cjcm/srch/dncdSrch?lang=en Canadian International Development Agency http://search-recherche.gc.ca/rGs/s_r?as_q=&as_epq=&as_oq=&as_eq=&1s_f3l2typ2=&as_nlo=&as_nlh&as_occt=&1s_s3t2s21rch=&1s_s4rt=&st1rt=0&st=a&num=10&langs=eng&cdn=cida Citizenship and Immigration Canada http://www.cic.gc.ca/search-recherche/index-eng.aspx Department of Finance http://www.fin.gc.ca/search-recherche/query-recherche-eng.aspx?t=a Department of Justice http://www.justice.gc.ca/eng/sch-rch/sch-rch.asp Department of National Defence http://www.index.forces.gc.ca/Srch.aspx?lang=en-CA&Scrn=Adv Environment Canada http://www.ec.gc.ca/default.asp?lang=En&n=ECD35C36 Fisheries and Oceans http://www.dfo-mpo.gc.ca/search-recherche-eng.htm Foreign Affairs and International Trade http://www.international.gc.ca/about-a_propos/search-recherche.aspx?lang=eng&view=d Heritage Canada http://www.pch.gc.ca/eng/1268230642921/1268230574484/s/q.s?S_SEARCH.language=eng&templateId=1&S_SFC.value=&S_SEARCH.parametricFields=&S_USES_PARAMETRIC.value=# Human Resources and Skills Development http://www3.hrsdc.gc.ca/search?site=hrsdc_en&client=hrsdc_wet_r12&output=xml_no_dtd&proxystylesheet=hrsdc_wet_r12&proxycustom=%3CHOME/%3E Industry Canada http://www.ic.gc.ca/eic/site/icgc.nsf/eng/06957.html?Open&q=%20&ieutf=%EF%BE%A0 Natural Resources Canada http://www2.nrcan.gc.ca/sr/index-eng.cfm Public Safety http://www.publicsafety.gc.ca/serv/srch/index-eng.aspx Public Works and Government Services Canada http://recherche-search.gc.ca/s_r?t3mpl1t34d=2&s5t34d=tpsgcpwgsc&l7c1l3=eng&S_F8LLT2XT=&S_m5m3typ3.sp3c5f53r=INDEX&S_m5m3typ3.t3xt6p3r1t7r=OR&S_m5m3typ3.v1l93=&S_S20RCH.p1r1m3tr5cF53lds=service,paudience&S_S20RCH.p1r1m3tr5cS7rt=documentcount&S_S20RCH.p1r1m3tr5cQ93ry=false&S_S2RV4C2.v1l93=&S_P08D42NC2.v1l93=&S_S20RCH.l1ng91g3=eng&S_d1t3fr7m.f53ld=documentdate&S_d1t3fr7m.d1t37p3r1t7r=gt&S_d1t3fr7m.v1l93=&S_d1t3t7.f53ld=documentdate&S_d1t3t7.d1t37p3r1t7r=lt&S_d1t3t7.v1l93=&S_08D4T.1ct57n=search&S_08D4T.s3rv5c3=advanced Service Canada http://recherche-search.gc.ca/s_r?t3mpl1t34d=2&s5t34d=service&l7c1l3=eng&S_08D4T.1ct57n=form&S_F8LLT2XT=&S_S20RCH.l1ng91g3=eng&S_08D4T.s3rv5c3=advanced Statistics Canada http://www.statcan.gc.ca/search-recherche/adv-ava-eng.htm Transport Canada http://search-recherche.tc.gc.ca/search.aspx?q=&cn-search-submit=Search Treasury Board http://www.tbs-sct.gc.ca/search-recherche/query-recherche-eng.aspx?t=a Note: Number of departments n = 21; Last accessed April 22nd, 2013 Table A3: Systems Review - List of Departments and Search Systems Reviewed ? Government of Canada 86 Table A4 Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Canada Facet/Filter Category Form of Use Score Audience Metadata standard 1 Contributor As filter by Department of National Defence 1 Coverage ? Spatial (Location) Metadata standard; as facet by Department of National Defence; as filter by Statistics Canada 2 Creator (Author) As filter by Department of National Defence 1 Date - extracted As filter by Department of National Defence 1 Date - modified As filter by Aboriginal Affairs and Northern Development, Department of Finance, Treasury Board, Service Canada, Agriculture and Agri-Food Canada, Public Works and Government Services Canada, Statistics Canada, Canadian International Development Agency, Department of National Defence 3 Date ? published (Date ? created) As filter by Department of National Defence 1 Date - reviewed As filter by Department of National Defence 1 Function (Activity) As facet by Department of National Defence 1 Item ? Extent (Size) As facet by Department of National Defence 1 Item ? Format (File Format) Metadata standard; as facet by Department of National Defence; as filter by Agriculture and Agri-Food Canada, Heritage Canada 2 Language As facet by Department of National Defence, Department of National Defence; as filter by Agriculture and Agri-Food Canada, Environment Canada, Natural Resources Canada, Heritage Canada; each department provides prominent link to switch between French and English 3 Source As facet by Department of National Defence 1 Subject (Topic) Metadata standard; as facet by Agriculture and Agri-Food Canada; as filter by Industry Canada, Department of Justice 2 Time Frame ? Coverage Temporal As filter by Aboriginal Affairs and Northern Development, Department of Finance, Treasury Board, Service Canada, Agriculture and Agri-Food Canada, Public Works and Government Services Canada, Statistics Canada, Canadian International Development Agency, Department of National Defence 3 Type - Document Metadata standard; as facet by Department of National Defence; as filter by Industry Canada, Canada Revenue Agency, Statistics Canada, Department of Justice, Human Resources and Skills Development, Heritage Canada 2 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. No facets or filter categories were found in the search systems of these departments: Overall Government Canada online search system, Foreign Affairs and International Trade , Public Safety , Citizenship and Immigration Canada , Transport Canada , Fisheries and Oceans Note: Number of departments n = 21; Status as of April 22nd, 2013 Table A4: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of Canada 87 Table A5 Systems Review - List of Departments and Search Systems Reviewed ? Government of the United Kingdom Department/Search System Link Attorney General?s Office https://www.gov.uk/government/organisations/attorney-generals-office Cabinet Office https://www.gov.uk/government/organisations/cabinet-office Department for Communities and Local Government https://www.gov.uk/government/organisations/department-for-communities-and-local-government Department for Culture, Media and Sport https://www.gov.uk/government/organisations/department-for-culture-media-sport Department for International Development https://www.gov.uk/government/organisations/department-for-international-development Department for Work & Pension http://search2.openobjects.com/kbroker/dwp/dwp/search/asearch.jsp Department of Business Innovation and Skills https://www.gov.uk/government/organisations/department-for-business-innovation-skills Department of Education http://www.education.gov.uk/search Department of Energy & Climate Change https://www.gov.uk/government/organisations/department-of-energy-climate-change Department of Environment, Food and Rural Affairs http://www.defra.gov.uk/ Department of Health https://www.gov.uk/government/organisations/department-of-health Department of Transport https://www.gov.uk/government/organisations/department-for-transport Department of Treasury http://www.hm-treasury.gov.uk/Search.aspx?terms=test Foreign & Commonwealth Office https://www.gov.uk/government/organisations/foreign-commonwealth-office GOV.UK single government website main page https://www.gov.uk/ GOV.UK single government website search system across https://www.gov.uk/search?q=test Home Office http://www.homeoffice.gov.uk/ Justice Department http://www.justice.gov.uk/search?collection=moj-matrix-dev-web&form=simple&profile=_default&query=test Ministry of Defense https://www.gov.uk/government/organisations/ministry-of-defence Northern Ireland Office https://www.gov.uk/government/organisations/northern-ireland-office Prime Ministers web presence http://www.number10.gov.uk/ Scotland Office https://www.gov.uk/government/organisations/scotland-office Wales Office https://www.gov.uk/government/organisations/wales-office Note: Number of departments n = 21; Last accessed April 22nd, 2013 Table A5: Systems Review - List of Departments and Search Systems Reviewed ? Government of the United Kingdom 88 Table A6 Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United Kingdom Facet/Filter Category Form of Use Score Accessibility Metadata standard 1 Addressee Metadata standard 1 Aggregation Level Metadata standard 1 Audience Metadata standard; as facet by Department of Education; as filter by Justice Department 2 Availability (location of object) Metadata standard 1 Contributor Metadata standard 1 Coverage - Spatial Metadata standard 1 Creator Metadata standard 1 Date - modified Metadata standard; as filter by Department for Work & Pension 1 Digital signature Metadata standard 1 Disposal Metadata standard 1 Item ? Format (Format) Metadata standard 1 Language Metadata standard 1 Mandate Metadata standard 1 Organization As filter by Justice Department 1 Preservation Metadata standard 1 Publisher Metadata standard 1 Relation Metadata standard 1 Rights - Access Rights Metadata standard 1 Source Metadata standard 1 Status Metadata standard 1 Subject (Topic) Metadata standard; as filter by main government web page; as facet by Department of Education; as filter by Justice Department 2 Time Frame As facet by Department of Education; as filter by Department for Work & Pension 2 Type - Document Metadata standard; as facet by Department of Education; as filter by Department of Treasury, Justice Department 2 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. No facets or filter categories were found in the search systems of these departments: Overall GOV.UK single government website search system, Attorney General?s Office, Cabinet Office, Department of Business Innovation and Skills, Department for Communities and Local Government, Department of Health, Department for International Development, Department for Culture, Media and Sport, Department of Transport, Department of Energy & Climate Change, Foreign & Commonwealth Office, Home Office, Ministry of Defense, Northern Ireland Office, Scotland Office, Wales Office Note: Number of departments n = 21; Status as of April 22nd, 2013 Table A6: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United Kingdom 89 Table A7 Systems Review - List of Departments and Search Systems Reviewed ? Government of the United States of America Department/Search System Link Department of Agriculture http://usdasearch.usda.gov/search/advanced?affiliate=usda&enable_highlighting=true&m=false&page=1&per_page=10 Department of Commerce http://search.commerce.gov/search?query=&op.x=0&op.y=0&affiliate=commerce.gov Department of Defense http://www.defense.gov/search/ Department of Education http://www.ed.gov/find Department of Energy http://energy.gov/search/site Department of Health and Human Services http://search.hhs.gov/search?q=&btnG=Search&site=HHS&entqr=3&ud=1&sort=date:D:L:d1&output=xml_no_dtd&ie=UTF-8&oe=UTF-8&lr=lang_en&client=HHS&proxystylesheet=HHS&ulang=en&ip=128.189.137.251&access=p&entqrm=0&proxycustom=%3CADVANCED/%3E Department of Homeland Security http://search.dhs.gov/search?query=&op=Search&affiliate=dhs Department of Housing and Urban Development http://search.usa.gov/search?affiliate=housingandurbandevelopment&query= Department of Justice http://searchjustice.usdoj.gov/search?client=default_frontend&proxystylesheet=default_frontend&proxycustom=%3CADVANCED/%3E Department of Labor http://www.dot.gov/gsearch Department of Labor - Browsing http://www.dol.gov/ Department of Labor - Search http://search.usa.gov/search?query=+&affiliate=u.s.departmentoflabor Department of State http://www.state.gov/# Department of the Interior http://search.usa.gov/search?affiliate=doi.gov&m=false&query= Department of Treasury http://search.treasury.gov/search?utf8=%E2%9C%93&sc=0&query=&m=&embedded=&affiliate=treasury&filter=moderate&commit=Search Department of Veterans Affairs http://www.index.va.gov/search/va/va_adv_search.jsp Government Printing Office?s Federal Digital System http://www.gpo.gov/fdsys/search/search.action?na=&se=&sm=&flr=&ercode=&dateBrowse=&govAuthBrowse=&collection=&historical=false&st=content%3A&psh=&sbh=&tfh=&originalSearch= USA.GOV search system http://search.usa.gov/search?utf8=%E2%9C%93&sc=0&query=&m=&embedded=&affiliate=usagov&filter=moderate&commit=Search White House http://search.whitehouse.gov/search?affiliate=wh&query=&submit.x=0&submit.y=0&form_id=usasearch_box Note: Number of departments n = 17; Last accessed April 22nd, 2013 Table A7: Systems Review - List of Departments and Search Systems Reviewed ? Government of the United States of America 90 Table A8 Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United States of America Facet/Filter Category Form of Use Score Audience As filter by Department of Labor 1 Availability (Location of object) As filter by Department of Labor 1 Coverage ? Spatial (Location about) As facet by Department of State, Government Printing Office?s Federal Digital System; As filter by Department of Labor 2 Creator (Author, Speaker) As facet by Department of State, Government Printing Office?s Federal Digital System 1 Date ? published As facet by Government Printing Office?s Federal Digital System 1 Item Format (File Type) As filter by Department of Agriculture, Department of Defense, Department of Education, Department of Health and Human Services, Department of Justice, Department of Veterans Affairs 2 Language As filter by Department of Education, Department of Health and Human Services, Department of Justice 2 Organization (agency, branch) As facet by Department of Veterans Affairs; as filter by USA.GOV search system, Department of Agriculture, Department of Commerce, Department of Defense, Department of Homeland Security, Department of Housing and Urban Development, Department of Labor, Department of the Interior, Department of Treasury, White House 3 Subject (Person - about, Topic) As facet by Department of Energy, Department of State, Government Printing Office?s Federal Digital System; as filter by Department of Labor 2 Time Frame As facet by Department of State, Government Printing Office?s Federal Digital System 1 Type - Aggregation Level (Collection) As facet by Government Printing Office?s Federal Digital System 1 Type ? Document As facet by Department of Energy, Department of State, Department of Veterans Affairs; as filter by USA.GOV search system, Department of Agriculture, Department of Commerce, Department of Defense, Department of Homeland Security, Department of Housing and Urban Development, Department of Labor, Department of the Interior, Department of Treasury, White House 3 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. No facets or filter categories were found in the search systems of these departments: Department of Energy Note: Number of departments n = 17; Status as of April 22nd, 2013 Table A8: Systems Review - Summary of Facets and Filter Categories Used in the Web Presence of the Government of the United States of America 91 Table A9 Systems Review - List of Departments Reviewed on Amazon.ca Department/Search System Link Baby http://www.amazon.ca/b/ref=sa_menu_allbaby?ie=UTF8&node=3561346011 Books http://www.amazon.ca/books-used-books-textbooks/b/ref=topnav_storetab_b?ie=UTF8&node=916520 Electronics http://www.amazon.ca/Electronics/b/ref=sa_menu_eva?ie=UTF8&node=667823011 Home & Garden http://www.amazon.ca/b/ref=sa_menu_allhomegard?ie=UTF8&node=2206275011 Movies & TV http://www.amazon.ca/dvds-used-dvd-boxed-sets/b/ref=sa_menu_mov?ie=UTF8&node=917972 Music http://www.amazon.ca/music-rock-classical-pop-jazz/b/ref=sa_menu_mu?_encoding=UTF8&node=916514 Software http://www.amazon.ca/software-business-education-finance-childrens/b/ref=sa_menu_sw?_encoding=UTF8&node=3198021 Sports & Outdoor http://www.amazon.ca/sporting-goods/b/ref=sa_menu_soa?ie=UTF8&node=2242989011 Tools & Building Supplies http://www.amazon.ca/Home-Improvement/b/ref=sa_menu_atools?_encoding=UTF8&node=3006902011 Video Games http://www.amazon.ca/video-games-hardware-accessories/b/ref=sa_menu_vg?ie=UTF8&node=3198031 Watches http://www.amazon.ca/b/ref=sa_menu_watches?ie=UTF8&node=2235620011 Note: Number of departments n = 11; Last accessed April 23rd, 2013 Table A9: Systems Review - List of Departments Reviewed on Amazon.ca 92 Table A10 Systems Review - Summary of Facets and Filter Categories Used on Amazon.ca Facet/Filter Category Form of Use Score Audience As facet by Baby, Watches 1 Availability As facet by Baby, Books, Home & Garden, Movies & TV, Software, Video Games, Watches 3 Creator As facet by Books 1 Date ? published As facet by Books, Movies & TV 1 Function As facet by Watches 1 Item ? Extent As facet by Electronics, Sports & Outdoor, Tools & Building Supplies, Watches 3 Item ? Format As facet by Books, Home & Garden, Movies & TV, Music, Tools & Building Supplies, Watches 2 Language As facet by Books 1 Organization As filter in all departments 3 Price As facet by Baby, Electronics, Home & Garden, Movies & TV, Software, Sports & Outdoor, Tools & Building Supplies, Video Games 3 Rating ? Source As facet by Baby, Electronics, Home & Garden, Sports & Outdoor, Tools & Building Supplies 2 Rating ? Item As facet by Baby, Electronics, Home & Garden, Sports & Outdoor, Tools & Building Supplies 2 Relation As facet by Electronics, Video Games 1 Source As facet by Baby, Electronics, Home & Garden, Software, Sports & Outdoor, Tools & Building Supplies, Watches 3 Special Attributes As facet by Movies & TV, Music 1 Terms & Conditions As facet by Baby, Books, Electronics, Home & Garden, Movies & TV, Music, Software, Tools & Building Supplies, Video Games, Watches 3 Type - Item As facet by Movies & TV, Music, Video Games 1 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. Note: Number of departments n = 11; Status as of April 22nd, 2013 Table A10: Systems Review - Summary of Facets and Filter Categories Used on Amazon.ca 93 Table A11 Systems Review - List of Departments Reviewed on eBay.ca Department Link Antiques http://antiques.shop.ebay.ca/ Automotive http://motors.shop.ebay.ca/ Books http://books.shop.ebay.ca/ Business & Industrial http://business.shop.ebay.ca/ Camera & Photo http://photography.shop.ebay.ca/ Cellphones & Accessories http://cell-phones.shop.ebay.ca/ Clothing, Shoes & Accessories http://clothing.shop.ebay.ca/ Coins & Paper Money http://coins.shop.ebay.ca/ Collectibles http://collectibles.shop.ebay.ca/ Computers/Tablets & Networking http://computers.shop.ebay.ca/ Consumer Electronics http://electronics.shop.ebay.ca/ Dolls & Bears http://dolls.shop.ebay.ca/ DVDs & Movies http://dvds.shop.ebay.ca/ Entertainment Memorabilia http://entertainment-memorabilia.shop.ebay.ca/ Gift Cards & Coupons http://www.ebay.ca/sch/?_sacat=172008 Health & Beauty http://healthbeauty.shop.ebay.ca/ Home & Garden http://home.shop.ebay.ca/ Jewelry & Watches http://jewelry.shop.ebay.ca/ Music http://music.shop.ebay.ca/ Musical Instruments & Gear http://www.ebay.ca/sch/?_sacat=619 Pet Supplies http://pet-supplies.shop.ebay.ca/ Pottery & Glass http://pottery.shop.ebay.ca/ Real Estate http://realestate.shop.ebay.ca/ Specialty Services http://www.ebay.ca/sch/?_sacat=316 Sporting Goods http://sporting-goods.shop.ebay.ca/ Sports Mem, Cards & Fan Shop http://sports-cards.shop.ebay.ca/ Stamps http://stamps.shop.ebay.ca/ Tickets http://www.ebay.ca/tickets/ Toys & Hobbies http://toys.shop.ebay.ca/ Travel http://travel.shop.ebay.ca/ Video Games & Consoles http://videogames.shop.ebay.ca/ Everything Else http://everythingelse.shop.ebay.ca/ Note: Number of departments n = 32; Last accessed April 23rd, 2013 Table A11: Systems Review - List of Departments Reviewed on eBay.ca 94 Table A12 Systems Review - Summary of Facets and Filter Categories Used on eBay.ca Facet/Filter Category Form of Use Score Audience As facet by Clothing, Shoes & Accessories, DVDs & Movies, Jewelry & Watches, Video Games & Consoles 1 Availability As facet by all departments/product categories 3 Coverage Spatial As facet by Travel 1 Creator As facet by Antiques, Automotive, Toys & Hobbies 1 Date ? available As facet by Tickets 1 Date ? published As facet by Antiques, Automotive, Collectibles 1 Item ? Extent As facet by Clothing, Shoes & Accessories, Home & Garden, Music, Pet Supplies, Real Estate, Travel 2 Item ? Format As facet by Antiques, Books, Business & Industrial, DVDs & Movies, Home & Garden, Jewelry & Watches, Music, Sports Mem, Cards & Fan Shop, Toys & Hobbies 2 Organization As facet or filter by all departments/product categories 3 Price As filter by all departments/product categories 3 Rating - Source Seller Rating, Seller Status 3 Relation As facet by Books, Computers/Tablets & Networking, Dolls & Bears, Toys & Hobbies, Video Games & Consoles 1 Rights - Access Rights As facet by Coins & Paper Money, Collectibles 1 Source As facet by Business & Industrial, Camera & Photo, Cellphones & Accessories, Clothing, Shoes & Accessories, Collectibles, Computers/Tablets & Networking, Consumer Electronics, Dolls & Bears, Gift Cards & Coupons, Health & Beauty, Home & Garden, Jewelry & Watches, Musical Instruments & Gear, Toys & Hobbies, Video Games & Consoles 3 Special Attributes As facet by Coins & Paper Money 1 Status As facet by all departments/product categories with some more specific options in Antiques, Coins & Paper Money, Collectibles, Sports Mem, Cards & Fan Shop, Toys & Hobbies, Video Games & Consoles 3 Subject As facet by Antiques, Books, Clothing, Shoes & Accessories, Sports Mem, Cards & Fan Shop, Tickets 1 Terms and Conditions As facet by all departments/product categories 3 Time Frame As facet by Travel 1 Type - Item As facet by Antiques, Coins & Paper Money, Computers/Tablets & Networking, DVDs & Movies, Gift Cards & Coupons, Home & Garden, Music, Musical Instruments & Gear, Real Estate, Specialty Services, Sports Mem, Cards & Fan Shop, Toys & Hobbies, Travel, Video Games & Consoles 2 Note: A facet or filter category is scored with a 1 if it is mentioned in system descriptions, standards, or guidelines for characteristics of content, or if it is made available for searchers in at least one of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 2 if it is made available for searchers in some of the sub organizations/areas covered by the reviewed organization. A facet or filter category is scored with a 3 if it is made available for searchers across the entire system or in a significant plurality of sub organizations/areas covered by the reviewed organization. Note: Number of departments n = 32; Status as of April 22nd, 2013 Table A12: Systems Review - Summary of Facets and Filter Categories Used on eBay.ca 95 Table A13 Systems Review - Assessment of Facets and Filter Categories Included in Questionnaire Facet/Filter Category GOV LIBINFO COM Total Comments Item ? Format 8 9 8 25 Use in questionnaire. Seems to be highly important. Use as \"Format of Object\". Type ? Item 9 9 7 25 Use in questionnaire. Is particularly interesting as it is related to genre theory discussed in the literature review. Use as \"Type of document\" in questionnaire as scenarios are based on information work tasks. Availability 3 9 9 21 Use in questionnaire. Date - published 3 12 5 20 Use in questionnaire. Different types of date should be treated as one if possible, hence use as \"Date created or published\". Audience 6 6 2 14 Use in questionnaire. Coverage - Spatial 6 6 1 13 Use in questionnaire. It has to be made clear to survey participants that this relates to being about a certain location, hence use as \"Geographical area (about)\". Terms and Conditions 9 9 Use in questionnaire. Initially seemed not very relevant to online content, due to referring to payment and shipping terms which are usually not relevant for online content. But it encompasses the concepts of license and rights holder information. Use as \"Terms of use\". Organization 9 9 Use in questionnaire. Use as \"Department (Organization)\". Rating - Source 8 8 Use in questionnaire. Rating of the seller or creator. Does only seem important in commercial systems. It is interesting to see whether users would find it useful across domains. Use as \"Rating of provider\". Time Frame 7 1 8 Use in questionnaire. Item ? Extent 2 4 6 Use in questionnaire. Use as \"Size\". Location - Source 3 3 Use in questionnaire. Location of object or organization, use as \"Geographical area (location)\" Date - available 1 1 2 Use in questionnaire. Although it could be the same as date published or created, it could also be significantly different and play an important role searchers? assessment of online content. Rating ? Item 2 2 Use in questionnaire. Similarly to rating of source interesting to look at across domains. Use as \"Rating of object\". Note: The figures for columns GOV, LIBINFO, and COM are based on the total score of 4 sample web presences in each of domains. Abbreviations used: COMM = Commercial Domain, GOV = Government Domain, LIBINFO = Library and Information Domain Table A13: Systems Review - Assessment of Facets and Filter Categories Included in Questionnaire 96 Table A14 Systems Review - Assessment of Facets and Filter Categories not Included in Questionnaire Facet/Filter Category GOV LIBINFO COM Total Comments Subject 8 12 4 24 Usable as facet and seems to be highly important. Creator 4 12 5 21 Itself not a facet, a type of creator, e.g. author, could function as facet. Language 7 12 1 20 Usable as facet, but as the study is mainly focused on content in the English language it does not seem necessary to include it. Source 3 6 6 15 It contains, similarly to creator, several types of facets, but is itself not a facet. Relation 2 9 2 13 Contains different kinds of facets Price 12 12 Only used in commercial settings, so would raise problems for participants when trying to generalize across domains. Organization 7 3 10 Contributor 3 6 9 Itself not a facet, a type of contributor, e.g. editor, could function as facet. Publisher 2 6 8 Special Attributes 3 5 8 This is an assortment of different particular attributes which difficult to conceptualize in a facet. Status 2 6 8 Accessibility 1 6 7 Not necessarily an issue when considering online content Date - modified 5 5 Type - Aggregation Level 2 3 5 Type - Service 1 3 4 Date - acquired 3 3 Function 2 1 3 Concept seems to be interesting but is rarely used. It is difficult to distinguish from Item - Type when only considering online content. Rights - Access Rights 2 1 3 Date - extracted 2 2 Mandate 2 2 Not usable as facet Addressee 1 1 Aggregation 1 1 Coverage - Jurisdiction 1 1 Can be seen as a special case of Coverage - Spatial. Date - Copyright 1 1 Date - issued 1 1 Date - licensed 1 1 Date - reviewed 1 1 Date - Validity 1 1 Digital signature 1 1 Disposal 1 1 Preservation 1 1 Note: The figures for columns GOV, LIBINFO, and COM are based on the total score of 4 sample web presences in each of the domains. Abbreviations used: COMM = Commercial Domain, GOV = Government Domain, LIBINFO = Library and Information Domain Table A14: Systems Review - Assessment of Facets and Filter Categories not Included in Questionnaire 97 Appendix B ? Study 1: Basic Participant Information Table A15 Study 1- Frequency of Age Ranges of Participants Frequency Percent Valid Percent Cumulative Percent 22 to 31 30 46.2 46.2 46.2 32 to 41 20 30.8 30.8 76.9 42 to 51 11 16.9 16.9 93.8 52 to 61 2 3.1 3.1 96.9 62 or older 2 3.1 3.1 100.0 Total 65 100.0 100.0 Table A15: Study 1 - Frequency of Age Ranges of Participants Table A16 Study 1 - Gender Distribution of Participants in Study Frequency Percent Valid Percent Cumulative Percent Female 39 60.0 60.0 60.0 Male 23 35.4 35.4 95.4 Other 1 1.5 1.5 96.9 Prefer not to tell 2 3.1 3.1 100.0 Total 65 100.0 100.0 Table A16: Study 1- Gender Distribution of Participants Table A17 Study 1 - Student Status of Participants Frequency Percent Valid Percent Cumulative Percent Yes, studying full time 31 47.7 47.7 47.7 Yes, studying part time 3 4.6 4.6 52.3 No, I am not currently undertaking formal study 31 47.7 47.7 100.0 Total 65 100.0 100.0 Table A17: Study 1 - Student Status of Participants Table A18 Study 1 - Employment Status of Participants Frequency Percent Valid Percent Cumulative Percent Yes, working full time 28 43.1 43.1 43.1 Yes, working part time 21 32.3 32.3 75.4 No, I am not currently employed. 16 24.6 24.6 100.0 Total 65 100.0 100.0 Table A18: Study 1 - Employment Status of Participants Table A19 Study 1 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) Frequency Percent Valid Percent Cumulative Percent Other 1 1.5 1.5 1.5 College, CEGEP or other non-university certificate or diploma 2 3.1 3.1 4.6 Bachelor's degree 7 10.8 10.8 15.4 Master's degree 43 66.2 66.2 81.5 Earned doctorate 12 18.5 18.5 100.0 Total 65 100.0 100.0 Table A19: Study 1 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) 98 Table A20 Study 1 - Self-Reported Skill Level of Participants in Searching the Internet Frequency Percent Valid Percent Cumulative Percent 4 2 3.1 3.1 3.1 5 16 24.6 24.6 27.7 6 27 41.5 41.5 69.2 7 (High) 20 30.8 30.8 100.0 Total 65 100.0 100.0 Table A20: Study 1 - Self-Reported Skill Level of Participants in Searching the Internet Table A21 Study 1 - Student Status of Participants in Age Group 22 to 31 Frequency Percent Valid Percent Cumulative Percent Yes, studying full time 20 66.7 66.7 66.7 Yes, studying part time 1 3.3 3.3 70.0 No, I am not currently undertaking formal study 9 30.0 30.0 100.0 Total 30 100.0 100.0 Table A21: Study 1 - Student Status of Participants in Age Group 22 to 31 99 Appendix C ? Statistics Canada CANSIM Excerpts Table A22 Statistics Canada - CANSIM Summary of Age Groups and Genders - Year 2012 Males Females Total Males Percentage Females Percentage Total Percentage 22 to 31 2,495,853 2,409,919 4,905,772 9.57 9.24 18.81 32 to 41 2,342,153 2,334,766 4,676,919 8.98 8.95 17.93 42 to 51 2,628,571 2,597,860 5,226,431 10.08 9.96 20.04 52 to 61 2,415,316 2,462,822 4,878,138 9.26 9.44 18.70 62 or older 2,912,849 3,483,770 6,396,619 11.17 13.36 24.52 Total 12,794,742 13,289,137 26,083,879 49.05 50.95 100.00 Footnotes: 1 Postcensal estimates are based on the 2006 Census counts adjusted for census net undercoverage, incompletely enumerated Indian reserves and for the estimated population growth that occurred since that census. Intercensal estimates are based on postcensal estimates and census counts adjusted for the censuses preceding and following the considered year. 2 Estimates are final intercensal up to 2005, final postcensal from 2006 to 2009, updated postcensal for 2010 and 2011 and preliminary postcensal for 2012. 6 Data for persons aged 90 to 100 years and over will be available from 2001. 7 The population growth, which is used to calculate population estimates, is comprised of the natural growth (CANSIM 51-0002 and 51-0013), international migration (CANSIM 51-0011) and interprovincial migration (CANSIM 51-0012). Source: Statistics Canada. Table 051-0001 - Estimates of population, by age group and sex for July 1, Canada, provinces and territories, annual (persons unless otherwise noted) (accessed: June 21, 2013) Table A22: Statistics Canada - CANSIM Summary of Age Groups and Genders - Year 201258 Table A23 Statistics Canada - Population 15 Years and Over by Highest Certificate, Diploma or Degree (2006 Census) Total Percentage Total - Highest certificate, diploma or degree1 25,664,220 100.00 No certificate, diploma or degree 6,098,325 23.76 Certificate, diploma or degree 19,565,895 76.24 High school certificate or equivalent2 6,553,420 25.54 Apprenticeship or trades certificate or diploma 2,785,420 10.85 College, CEGEP or other non-university certificate or diploma3 4,435,140 17.28 University certificate, diploma or degree 5,791,915 22.57 University certificate or diploma below bachelor level 1,136,145 4.43 University certificate or degree 4,655,770 18.14 Bachelor's degree 2,981,465 11.62 University certificate or diploma above bachelor level 493,540 1.92 Degree in medicine, dentistry, veterinary medicine or optometry 136,845 0.53 Master's degree 866,975 3.38 Earned doctorate 176,945 0.69 Footnotes: 1. Total - Highest certificate, diploma or degree: 'Highest certificate, diploma or degree' refers to the highest certificate, diploma or degree completed based on a hierarchy which is generally related to the amount of time spent 'in-class'. For postsecondary completers, a university education is considered to be a higher level of schooling than a college education, while a college education is considered to be a higher level of education than in the trades. Although some trades requirements may take as long or longer to complete than a given college or university program, the majority of time is spent in on-the-job paid training and less time is spent in the classroom. 2. High school certificate or equivalent: 'High school certificate or equivalent' includes persons who have graduated from a secondary school or equivalent. Excludes persons with a postsecondary certificate, diploma or degree. Examples of postsecondary institutions include community colleges, institutes of technology, CEGEPs, private trade schools, private business colleges, schools of nursing and universities. 3. College, CEGEP or other non-university certificate or diploma: 'College, CEGEP or other non-university certificate or diploma' replaces the category 'Other non-university certificate or diploma' in previous censuses. This category includes accreditation by non-degree-granting institutions such as community colleges, CEGEPs, private business colleges and technical institutes. Source: Statistics Canada, Census of Population. Last modified: 2009-10-06. Table A23: Statistics Canada - Population 15 Years and Over by Highest Certificate, Diploma or Degree (2006 Census)59 58 http://www5.statcan.gc.ca/cansim/a26?lang=eng&retrLang=eng&id=0510001 59 http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/educ43a-eng.htm 100 Table A24 Statistics Canada - CANSIM Summary of Student Status for Age Group 15 to 29 Years - Year 2012 Total Percentage Students 3116.4 45.62 Full-time students (11) 2819.8 41.28 Part-time students (12) 296.6 4.34 Non-students (13) 3714.3 54.38 Total 6830.7 100.00 Footnotes: 11 People enrolled full-time at an educational institution. 12 People enrolled part-time at an educational institution. 13 People not enrolled in any educational institutions 14 Estimates in this table are based on an 8-month average for the calendar year (i.e. January to April and September to December). 15 The Labour force survey collection of tables, starting with number 282-, is large with many possible cross-tabulations for the 10 provinces and other geographic regions. To ensure respondent's confidentiality, detailed data are suppressed. Data for Canada, Quebec, Ontario, Alberta and British Columbia are suppressed if the estimate is below 1,500, for Newfoundland and Labrador, Nova Scotia, New Brunswick, Manitoba and Saskatchewan, if the estimate is below 500, and for Prince Edward Island, under 200. For suppression levels within census metropolitan areas (CMAs) and economic regions (ERs), use the respective provincial suppression levels above. While suppressing to protect respondent confidentiality has the added effect of blocking-out the lowest-quality LFS data, some remaining non-suppressed data in these very large LFS CANSIM tables may be of insufficient quality to allow for accurate interpretation. Please be warned that the more detailed your LFS CANSIM download, the smaller the sample size upon which your LFS estimates will be based, and the greater the risk of downloading poorer quality data. 16 Estimates prior to 1996 are based on 2001 census population counts, while estimates from 1996 onwards are based on 2006 census population counts. Source: Statistics Canada. Table 282-0095 - Labour force survey estimates (LFS), by full- and part-time students during school months, sex and age group, annual (persons unless otherwise noted) (accessed: June 21, 2013) Table A24: Statistics Canada - CANSIM Summary of Student Status for Age Group 15 to 29 Years - Year 201260 Table A25 Statistics Canada - CANSIM Summary of Employment Status for Age Group 15 Years and Above - Year 2012 Total Percentage Employment (3) 17,507.7 92.75 Full-time employment (4) 14,212.9 75.30 Part-time employment (5) 3,294.8 17.45 Unemployment (6) 1,368.4 7.25 Total 18,876.1 100.00 Footnotes: 3 Number of persons who, during the reference week, worked for pay or profit, or performed unpaid family work or had a job but were not at work due to own illness or disability, personal or family responsibilities, labour dispute, vacation, or other reason. Those persons on layoff and persons without work but who had a job to start at a definite date in the future are not considered employed. Estimates in thousands, rounded to the nearest hundred. 4 Full-time employment consists of persons who usually work 30 hours or more per week at their main or only job. Estimates in thousands, rounded to the nearest hundred. 5 Part-time employment consists of persons who usually work less than 30 hours per week at their main or only job. Estimates in thousands, rounded to the nearest hundred. 6 Number of persons who, during the reference week, were without work, had actively looked for work in the past four weeks, and were available for work. Those persons on layoff or who had a new job to start in four weeks or less are considered unemployed. Estimates in thousands, rounded to the nearest hundred. 11 The Labour force survey collection of tables, starting with number 282-, is large with many possible cross-tabulations for the 10 provinces and other geographic regions. To ensure respondent's confidentiality, detailed data are suppressed. Data for Canada, Quebec, Ontario, Alberta and British Columbia are suppressed if the estimate is below 1,500, for Newfoundland and Labrador, Nova Scotia, New Brunswick, Manitoba and Saskatchewan, if the estimate is below 500, and for Prince Edward Island, under 200. For suppression levels within census metropolitan areas (CMAs) and economic regions (ERs), use the respective provincial suppression levels above. While suppressing to protect respondent confidentiality has the added effect of blocking-out the lowest-quality LFS data, some remaining non-suppressed data in these very large LFS CANSIM tables may be of insufficient quality to allow for accurate interpretation. Please be warned that the more detailed your LFS CANSIM download, the smaller the sample size upon which your LFS estimates will be based, and the greater the risk of downloading poorer quality data. 12 Estimates prior to 1996 are based on 2001 census population counts, while estimates from 1996 onwards are based on 2006 census population counts. Source: Statistics Canada. Table 282-0002 - Labour force survey estimates (LFS), by sex and detailed age group, annual (persons unless otherwise noted) (accessed: June 21, 2013) Table A25: Statistics Canada - CANSIM Summary of Employment Status for Age Group 15 Years and Above - Year 201261 60http://www5.statcan.gc.ca/cansim/a26?lang=eng&retrLang=eng&id=2820095 61 http://www5.statcan.gc.ca/cansim/a26?lang=eng&retrLang=eng&id=2820002 101 Appendix D ? Study 1: Questionnaire Example Activity-Based Discovery of Online Content Summary of questionnaire This questionnaire is part of a study investigating how individuals discover online content. Completing the questionnaire should take 5 to 10 minutes. This research is performed as part of a graduate thesis on the use of search filters. Please note that we are looking for participants who are: ? 22 years or older ? citizens or permanent residents of Canada ? experienced in using the Web for discovering online content This questionnaire contains five 5 brief pages of questions. Here is an overview for each page: ? Page 1: Evaluation of filter categories for finding a piece of information you know exists ? Page 2: Evaluation of filter categories for finding information to help you accomplish something ? Page 3: Evaluation of filter categories for learning about a new area of knowledge ? Page 4: Questions regarding your perception of using filter categories ? Page 5: Demographic questions The responses to the questionnaire will be reported without any reference to you specifically. All information that you provide will be treated confidentially and your identity will not be revealed in reporting the study results. After completing the questionnaire you will be eligible to enter a draw for a Samsung Galaxy Tab 2 (7-Inch, Wi-Fi). Entering the draw will not undermine the anonymity and confidentiality of your survey responses. If you have any questions, please read the detailed consent form or contact kristof.kessler@diigubc.ca. I have read the explanation about this study. If I complete the questionnaire, it will be assumed that my consent has been given. However, I realize that my participation is voluntary and that I am free to withdraw from the study at any time. To give you a better understanding of the kinds of search filters you will be asked about, below are two examples of websites that use search filters: Amazon.ca and the Vancouver Public Library catalogue. 102 Source: http://www.amazon.ca Source: http://www.vpl.ca Who can you contact if you have complaints or concerns about the study? If you have any concerns about your rights as a research subject and/or your experiences while participating in this study, you may contact the Research Subject Information Line in the UBC Office of Research Services at 604-822-8598 or if long distance e-mail RSIL@ors.ubc.ca or call toll free 1-877-822-8598. Please click on the Next Page button to go to page 1. Thanks a lot! 103 Page 1 of 5 Type of Task: Finding a piece of information you know exists. The example scenarios below are provided to give you a better understanding of what real-world situations could fall under this type of task. ? Known-Item Scenario 1: You want to start filing your taxes for 2012 and you are seeking the guidelines on how to do this. ? Known-Item Scenario 2: You are seeking a list with quality readings about wildlife in South-East Asia. ? Known-Item Scenario 3: You decided to purchase a new car, but you cannot decide between two models. You are seeking reviews about the car models. Considering this type of task, please answer the following question: 1) If your task is to find a piece of information you know exists and the search engine would allow you to filter content, how would you rate the usefulness of the filter categories on the left-hand side for finding helpful content on a scale from 1 to 7? 1 (Low) 2 3 4 5 6 7 (High) Audience, e.g. the document is for Seniors, the toy is for children over 3 ? ? ? ? ? ? ? Availability, e.g. the item is available for purchase or use; the item is available after registering ? ? ? ? ? ? ? Date available, e.g. release date, date made available to the public online ? ? ? ? ? ? ? Date created or published ? ? ? ? ? ? ? Department (Organization), e.g. Environment Canada, Home Furnishings Dept. ? ? ? ? ? ? ? Format of object, e.g. paperback, pdf ? ? ? ? ? ? ? Geographical area (about), e.g. the document is about British Columbia or France ? ? ? ? ? ? ? Geographical area (location), e.g. the item is located in Manhattan or Soho ? ? ? ? ? ? ? Ratings of object, e.g. reader?s satisfaction with a book, four star rating of a smartphone ? ? ? ? ? ? ? Ratings of provider, e.g. satisfaction of buyers with seller, satisfaction of client with agent ? ? ? ? ? ? ? Size, e.g. the document has over 100 pages, the car seats 5 passengers ? ? ? ? ? ? ? Terms of use, e.g. do not share, do not resell, may alter ? ? ? ? ? ? ? Timeframe, e.g. the document is about the 19th century, the event occurs in May ? ? ? ? ? ? ? Type of document, e.g., resource list, FAQ, report ? ? ? ? ? ? ? 104 Page 2 of 5 Type of Task: Finding information to help you accomplish something. The example scenarios below are provided to give you a better understanding of what real-world situations could fall under this type of task. ? Doing Scenario 1: An elderly uncle has had a stroke and is confined to a wheelchair, but he and your aunt want to continue to live in their own home. You are seeking information how to adapt their home to the new circumstances. ? Doing Scenario 2: You want to write a well-prepared letter to the editor in response to a news article you read in the Globe and Mail, and are looking for guidance. ? Doing Scenario 3: You are planning to import a recently bought car from the U.S. to Canada and are looking for guidance. Considering this type of task, please answer the following question: 2) If your task is to find information to help you accomplish something and the search engine would allow you to filter content, how would you rate the usefulness of the filter categories on the left-hand side for finding helpful content on a scale from 1 to 7? 1 (Low) 2 3 4 5 6 7 (High) Audience, e.g. the document is for Seniors, the toy is for children over 3 ? ? ? ? ? ? ? Availability, e.g. the item is available for purchase or use; the item is available after registering ? ? ? ? ? ? ? Date available, e.g. release date, date made available to the public online ? ? ? ? ? ? ? Date created or published ? ? ? ? ? ? ? Department (Organization), e.g. Environment Canada, Home Furnishings Dept. ? ? ? ? ? ? ? Format of object, e.g. paperback, pdf ? ? ? ? ? ? ? Geographical area (about), e.g. the document is about British Columbia or France ? ? ? ? ? ? ? Geographical area (location), e.g. the item is located in Manhattan or Soho ? ? ? ? ? ? ? Ratings of object, e.g. reader?s satisfaction with a book, four star rating of a smartphone ? ? ? ? ? ? ? Ratings of provider, e.g. satisfaction of buyers with seller, satisfaction of client with agent ? ? ? ? ? ? ? Size, e.g. the document has over 100 pages, the car seats 5 passengers ? ? ? ? ? ? ? Terms of use, e.g. do not share, do not resell, may alter ? ? ? ? ? ? ? Timeframe, e.g. the document is about the 19th century, the event occurs in May ? ? ? ? ? ? ? Type of document, e.g., resource list, FAQ, report ? ? ? ? ? ? ? 105 Page 3 of 5 Type of task: Learning about a new area of knowledge. The example scenarios below are provided to give you a better understanding of what real-world situations could fall under this type of task. ? Learning Scenario 1: You want to learn about the weather phenomena El Ni?a and El Ni?o, and how they impact on weather patterns in different regions across Canada. ? Learning Scenario 2: You are interested in learning about the history of British Columbia and want to find appropriate material. ? Learning Scenario 3: You are planning to buy a new laptop and want to make sure you are choosing a model which offers good quality and features at a reasonable price. Considering this type of task, please answer the following question: 3) If your task is to learn about a new area of knowledge and the search engine would allow you to filter content, how would you rate the usefulness of the filter categories on the left-hand side for finding helpful content on a scale from 1 to 7? 1 (Low) 2 3 4 5 6 7 (High) Audience, e.g. the document is for Seniors, the toy is for children over 3 ? ? ? ? ? ? ? Availability, e.g. the item is available for purchase or use; the item is available after registering ? ? ? ? ? ? ? Date available, e.g. release date, date made available to the public online ? ? ? ? ? ? ? Date created or published ? ? ? ? ? ? ? Department (Organization), e.g. Environment Canada, Home Furnishings Dept. ? ? ? ? ? ? ? Format of object, e.g. paperback, pdf ? ? ? ? ? ? ? Geographical area (about), e.g. the document is about British Columbia or France ? ? ? ? ? ? ? Geographical area (location), e.g. the item is located in Manhattan or Soho ? ? ? ? ? ? ? Ratings of object, e.g. reader?s satisfaction with a book, four star rating of a smartphone ? ? ? ? ? ? ? Ratings of provider, e.g. satisfaction of buyers with seller, satisfaction of client with agent ? ? ? ? ? ? ? Size, e.g. the document has over 100 pages, the car seats 5 passengers ? ? ? ? ? ? ? Terms of use, e.g. do not share, do not resell, may alter ? ? ? ? ? ? ? Timeframe, e.g. the document is about the 19th century, the event occurs in May ? ? ? ? ? ? ? Type of document, e.g., resource list, FAQ, report ? ? ? ? ? ? ? 106 Page 4 of 5 4) Here are some additional filter categories. Thinking of the types of tasks in the questionnaire, would you consider them useful? Yes No Date modified ? ? Function of object ? ? Language ? ? Price ? ? Publisher ? ? Subject ? ? Terms and Conditions ? ? 5) Can you think of any other filter categories that could have been useful considering the activities you encountered in the previous pages of the survey? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ 6) Please describe any difficulty you encountered in rating the usefulness of filter categories: ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ 7) What are your general impressions of using filters when searching online? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ 8) When do you consider filters most useful? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ 107 Page 5 of 5 9) How did you find out about this study? ? Facebook advertisement ? Poster advertisement ? Other (please specify) ______________________________________ 10) Please indicate your age: ? 21 or younger ? 22 to 31 ? 32 to 41 ? 42 to 51 ? 52 to 61 ? 62 or older 11) Please indicate your gender: ? Female ? Male ? Other ? Prefer not to tell 12) Are you currently a student? ? Yes, studying full time ? Yes, studying part time ? No, I am not currently undertaking formal study 13) Are you currently employed? ? Yes, working full time ? Yes, working part time ? No, I am not currently employed. 14) What is the highest degree or level of school you have completed? If currently enrolled, mark the program or degree that is in progress. ? Public or high school, no diploma ? High school diploma or equivalent ? Apprenticeship or trades certificate or diploma ? College, CEGEP or other non-university certificate or diploma ? Bachelor's degree ? Degree in medicine, dentistry, veterinary medicine or optometry ? Master's degree ? Earned doctorate ? Other (please specify) _________________________________ 15) Please indicate your status in Canada: ? Canadian Citizen 108 ? Permanent Resident of Canada ? Other (please specify) ____________________________________ 16) In terms of searching the Internet how do you rate your skill level on a scale from 1 to 7? ? 1 (Low) ? 2 ? 3 ? 4 ? 5 ? 6 ? 7 (High) 17) Do you have any further comments about the survey? If so, please specify. ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ Thank you for your participation in this research! After submitting this survey you will have the opportunity to enter a draw for a $50 gift certificate (chance of 1 in 200). You will also have the opportunity to indicate your interest in receiving the results of this research and in participating in further research to this topic. Enter Draw after Participation in Survey Thank you again for participating in the survey Activity-Based Discovery of Online Content. 1) Please enter your email address if you want to enter the draw for a $50 online gift certificate and/or if you want to be considered for participation in future studies: ____________________________________________________________ 2) Do you want to enter the draw? ? Yes ? No 3) Are you living in Vancouver (BC) and would like to be considered for participation in future studies? ? Yes ? No 109 Appendix E ? Study 1: Descriptive Statistics, Normality, and Variance Details Table A26 Study 1- Descriptive Statistics for Assessment of Perceived Usefulness of Facets N Mini- mum Maxi- mum Median Mean Std. Deviation Skewness Kurtosis Statistic Std. Error Statistic Std. Error Cross-Task - Audience 83 1.00 7.00 3.85 1.472 -.102 .264 -.745 .523 Cross-Task - Availability 83 1.67 7.00 4.49 1.535 -.039 .264 -.883 .523 Cross-Task - Date available 83 1.00 7.00 4.44 1.651 -.119 .264 -.802 .523 Cross-Task - Date created or published 83 2.00 7.00 5.31 1.332 -.354 .264 -.749 .523 Cross-Task - Department (Organization) 82 1.00 7.00 5.41 1.231 -.813 .266 1.178 .526 Cross-Task - Format of object 83 1.00 7.00 4.82 1.478 -.582 .264 .004 .523 Cross-Task - Geographical area (about) 83 1.00 7.00 5.22 1.182 -.880 .264 1.482 .523 Cross-Task - Geographical area (location) 83 1.00 7.00 4.36 1.482 -.213 .264 -.566 .523 Cross-Task - Ratings of object 82 1.00 7.00 4.46 1.429 -.753 .266 .126 .526 Cross-Task - Ratings of provider 83 1.00 7.00 4.33 1.517 -.301 .264 -.558 .523 Cross-Task - Size 83 1.00 7.00 4.45 1.407 -.367 .264 -.490 .523 Cross-Task - Terms of use 83 1.00 7.00 3.36 1.573 .267 .264 -.848 .523 Cross-Task - Timeframe 83 1.00 7.00 5.17 1.274 -.593 .264 .468 .523 Cross-Task - Type of document 83 1.33 7.00 5.36 1.248 -.434 .264 -.232 .523 Doing - Audience 76 1.00 7.00 5.00 4.17 1.754 -.390 .276 -.912 .545 Doing - Availability 76 1.00 7.00 5.00 4.57 1.907 -.394 .276 -.959 .545 Doing - Date available 74 1.00 7.00 4.00 4.11 1.941 .028 .279 -1.131 .552 Doing - Date created or published 75 1.00 7.00 5.00 4.83 1.870 -.314 .277 -1.107 .548 Doing - Department (Organization) 75 1.00 7.00 5.00 5.20 1.507 -.448 .277 -.456 .548 Doing - Format of object 76 1.00 7.00 5.00 4.76 1.917 -.550 .276 -.761 .545 Doing - Geographical area (about) 75 2.00 7.00 5.00 4.96 1.370 -.250 .277 -.487 .548 Doing - Geographical area (location) 76 1.00 7.00 4.00 4.26 1.739 -.262 .276 -.767 .545 Doing - Ratings of object 72 1.00 7.00 5.00 4.57 1.767 -.386 .283 -.796 .559 Doing - Ratings of provider 75 1.00 7.00 4.00 4.35 1.656 -.245 .277 -.726 .548 Doing - Size 75 1.00 7.00 5.00 4.29 1.799 -.240 .277 -.927 .548 Doing - Terms of use 76 1.00 7.00 3.00 3.45 1.829 .248 .276 -1.042 .545 Doing - Timeframe 75 2.00 7.00 5.00 4.87 1.554 -.172 .277 -.913 .548 Doing - Type of document 76 2.00 7.00 6.00 5.32 1.499 -.635 .276 -.593 .545 Known-Item - Audience 70 1.00 7.00 3.00 3.53 1.961 .217 .287 -1.175 .566 Known-Item - Availability 69 1.00 7.00 4.00 4.43 1.974 -.255 .289 -1.097 .570 Known-Item - Date available 69 1.00 7.00 5.00 4.97 1.902 -.618 .289 -.755 .570 Known-Item - Date created or published 69 1.00 7.00 6.00 5.46 1.623 -1.134 .289 .532 .570 Known-Item - Department (Organization) 69 2.00 7.00 6.00 5.68 1.312 -.872 .289 .219 .570 Known-Item - Format of object 67 1.00 7.00 5.00 4.79 1.610 -.457 .293 -.467 .578 Known-Item - Geographical area (about) 67 1.00 7.00 6.00 5.31 1.469 -.862 .293 .179 .578 Known-Item - Geographical area (location) 69 1.00 7.00 5.00 4.38 1.783 -.128 .289 -.956 .570 Known-Item - Ratings of object 69 1.00 7.00 5.00 4.30 2.110 -.260 .289 -1.307 .570 Known-Item - Ratings of provider 69 1.00 7.00 5.00 4.32 2.076 -.207 .289 -1.350 .570 Known-Item - Size 69 1.00 7.00 5.00 4.45 1.787 -.297 .289 -.756 .570 Known-Item - Terms of use 69 1.00 7.00 3.00 3.19 1.865 .487 .289 -.850 .570 Known-Item - Timeframe 69 1.00 7.00 5.00 5.14 1.537 -.651 .289 -.117 .570 Known-Item - Type of document 69 1.00 7.00 6.00 5.49 1.587 -1.021 .289 .625 .570 Learning - Audience 75 1.00 7.00 4.00 3.93 1.796 -.142 .277 -1.146 .548 Learning - Availability 75 1.00 7.00 5.00 4.56 1.803 -.221 .277 -1.193 .548 Learning - Date available 74 1.00 7.00 5.00 4.66 1.896 -.363 .279 -1.034 .552 Learning - Date created or published 75 2.00 7.00 6.00 5.53 1.464 -.678 .277 -.480 .548 Learning - Department (Organization). 73 1.00 7.00 6.00 5.45 1.395 -.925 .281 .739 .555 Learning - Format of object 75 1.00 7.00 5.00 4.79 1.655 -.495 .277 -.379 .548 Learning - Geographical area (about) 75 1.00 7.00 6.00 5.49 1.510 -1.088 .277 .900 .548 Learning - Geographical area (location) 75 1.00 7.00 5.00 4.47 1.862 -.404 .277 -.878 .548 Learning - Ratings of object 74 1.00 7.00 5.00 4.58 1.553 -.663 .279 -.380 .552 Learning - Ratings of provider 75 1.00 7.00 5.00 4.35 1.656 -.410 .277 -.697 .548 Learning - Size 74 1.00 7.00 5.00 4.58 1.508 -.357 .279 -.538 .552 Learning - Terms of use 74 1.00 7.00 3.00 3.42 1.843 .314 .279 -.943 .552 Learning - Timeframe 74 1.00 7.00 6.00 5.55 1.366 -.994 .279 .765 .552 Learning - Type of document 75 1.00 7.00 6.00 5.40 1.533 -.753 .277 -.215 .548 Table A26: Study 1 - Descriptive Statistics for Assessment of Perceived Usefulness of Facets 110 Table A27 Study 1 - tests of Normality for Perceived Usefulness Score Type of Task Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Perceived Usefulness Doing .146 1052 .000 .929 1052 .000 Learning .172 1043 .000 .915 1043 .000 Known-Item .161 963 .000 .905 963 .000 a. Lilliefors Significance Correction Table A27: Study 1 - tests of Normality for Perceived Usefulness Score Table A28 Study 1 - Independent-Samples Kruskal-Wallis test Statistics for Perceived Usefulness with Grouping Variable Type of Task Doing Known-Item Learning Content Characteristic N Mean Rank N Mean Rank N Mean Rank Chi- square df Asymp. Sig. Decision Audience 76 120.81 70 98.79 75 112.46 4.501 2 .105 Retain the null hypothesis Availability 76 112.18 69 108.17 75 110.95 .153 2 .926 Retain the null hypothesis Date available 74 94.13 69 122.14 74 111.62 7.507 2 .023 Reject the null hypothesis Date created or published 75 95.82 69 116.7 75 118.02 6.039 2 .049 Reject the null hypothesis Department (Organization) 75 98.99 69 119.28 73 109.57 3.985 2 .136 Retain the null hypothesis Format of object 76 110.96 67 108.58 75 108.84 .065 2 .968 Retain the null hypothesis Geographical area (about) 75 93.94 67 112.36 75 121.06 7.632 2 .022 Reject the null hypothesis Geographical area (location) 76 106.51 69 110.26 75 114.77 .654 2 .721 Retain the null hypothesis Ratings of object 72 110.02 69 104.38 74 109.41 .358 2 .836 Retain the null hypothesis Ratings of provider 75 109.26 69 111.08 75 109.75 .032 2 .984 Retain the null hypothesis Size 75 104.89 69 109.92 74 113.78 .770 2 .681 Retain the null hypothesis Terms of use 76 113.41 69 103.88 74 112.2 .978 2 .613 Retain the null hypothesis Timeframe 75 96.01 69 107.96 74 124.6 8.054 2 .018 Reject the null hypothesis Type of document 76 105.99 69 115.55 75 110.43 .863 2 .649 Retain the null hypothesis All facets 1052 1467.27 963 1547.33 1043 1575.81 8.375 2 .013 Reject the null hypothesis Null Hypothesis (significance level=0.05): The distribution of Perceived Useful is the same across categories of Type of Task. Table A28: Study 1 - Independent-Samples Kruskal-Wallis test Statistics for Perceived Usefulness with Grouping Variable Type of Task Table A29 Study 1 - Content Characteristic Date Available: Mann Whitney U test for Pairwise Comparison Between Types of Tasks Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Doing - Known-Item -17.493 10.180 -1.718 .860 .257 Doing - Learning -28.009 10.363 -2.703 .007 .021 Known-Item - Learning -10.546 10.363 -1.015 .310 .931 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A29: Study 1 - Content Characteristic Date Available: Mann Whitney U test for Pairwise Comparison Between Types of Tasks Table A30 Study 1 - Content Characteristic Date Created or Published: Mann Whitney U test for Pairwise Comparison Between Types of Tasks Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Doing - Known-Item -20.876 10.294 -2.028 0.043 0.128 Doing - Learning -22.200 10.077 -2.203 0.023 0.083 Known-Item - Learning 1.324 10.294 0.129 0.898 1.000 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A30: Study 1 - Content Characteristic Date Created or Published: Mann Whitney U test for Pairwise Comparison Between Types of Tasks 111 Table A31 Study 1 - Content Characteristic Geographical Area (about): Mann Whitney U test for Pairwise Comparison Between Types of Tasks Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Doing - Known-Item -18.418 10.304 -1.788 0.074 0.222 Doing - Learning -27.120 10.009 -2.709 0.007 0.020 Known-Item - Learning 8.702 10.304 0.845 0.398 1.000 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A31: Study 1 - Content Characteristic Geographical Area (about): Mann Whitney U test for Pairwise Comparison Between Types of Tasks Table A32 Study 1 - Content Characteristic Timeframe: Mann Whitney U test for Pairwise Comparison Between Types of Tasks Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Doing - Known-Item -11.950 10.295 -1.161 0.246 0.737 Doing - Learning -28.588 10.113 -2.827 0.005 0.014 Known-Item - Learning 16.638 10.329 1.611 0.107 0.322 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A32: Study 1 - Content Characteristic Timeframe: Mann Whitney U test for Pairwise Comparison Between Types of Tasks 112 Appendix F ? Study 2: Basic Participants Information Table A33 Study 2 - Frequency of Age Ranges of Participants Frequency Percent Valid Percent Cumulative Percent 22 to 31 15 75.0 75.0 75.0 32 to 41 4 20.0 20.0 95.0 42 to 51 1 5.0 5.0 100.0 Total 20 100.0 100.0 Table A33: Study 2 - Frequency of Age Ranges of Participants Table A34 Study 2 - Gender Distribution of Participants Frequency Percent Valid Percent Cumulative Percent Female 13 65.0 65.0 65.0 Male 7 35.0 35.0 100.0 Total 20 100.0 100.0 Table A34: Study 2 - Gender Distribution of Participants Table A35 Study 2 - Student Status of Participants Frequency Percent Valid Percent Cumulative Percent Yes, studying full time 15 75.0 75.0 75.0 Yes, studying part time 2 10.0 10.0 85.0 No, I am not currently undertaking formal study 3 15.0 15.0 100.0 Total 20 100.0 100.0 Table A35: Study 2 - Student Status of Participants Table A36 Study 2 - Employment Status of Participants Frequency Percent Valid Percent Cumulative Percent Yes, working full time 1 5.0 5.0 5.0 Yes, working part time 11 55.0 55.0 60.0 No, I am not currently employed. 8 40.0 40.0 100.0 Total 20 100.0 100.0 Table A36: Study 2 - Employment Status of Participants Table A37 Study 2 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) Frequency Percent Valid Percent Cumulative Percent Bachelor's degree 8 40.0 40.0 40.0 Master's degree 10 50.0 50.0 90.0 Doctorate 2 10.0 10.0 100.0 Total 20 100.0 100.0 Table A37: Study 2 - Academic Degree Status of Participants (Highest Degree Earned or in Progress) 113 Table A38 Study 2 - Self-Reported Skill Level of Participants in Searching the Internet Frequency Percent Valid Percent Cumulative Percent 4 1 5.0 5.0 5.0 5 9 45.0 45.0 50.0 6 7 35.0 35.0 85.0 7 3 15.0 15.0 100.0 Total 20 100.0 100.0 Table A38: Study 2 - Self-Reported Skill Level of Participants in Searching the Internet Table A39 Study 2 - Student Status of Participants in Age Group 22 to 31 Frequency Percent Valid Percent Cumulative Percent Yes, studying full time 11 73.3 73.3 73.3 Yes, studying part time 2 13.3 13.3 86.7 No, I am not currently undertaking formal study 2 13.3 13.3 100.0 Total 15 100.0 100.0 Table A39: Study 2 - Student Status of Participants in Age Group 22 to 31 114 Appendix G ? Study 2: System and Activity Assignment Table A40 Study 2 - System and Activity Assignment Participant# System Scenario 1 Scenario 2 Scenario 3 1* Experimental System A to Z Doing Known-Item Learning 2* Baseline System Known-Item Learning Doing 3** Experimental System Z to A Learning Doing Known-Item 4 Baseline System Doing Learning Known-Item 5 Experimental System A to Z Known-Item Doing Learning 6 Baseline System Learning Known-Item Doing 7 Experimental System Z to A Doing Known-Item Learning 8 Baseline System Known-Item Learning Doing 9 Experimental System A to Z Learning Doing Known-Item 10 Baseline System Doing Learning Known-Item 11 Experimental System Z to A Known-Item Doing Learning 12 Baseline System Learning Known-Item Doing 13 Experimental System A to Z Doing Known-Item Learning 14 Baseline System Known-Item Learning Doing 15 Experimental System Z to A Learning Doing Known-Item 16 Baseline System Doing Learning Known-Item 17 Experimental System A to Z Known-Item Doing Learning 18 Baseline System Learning Known-Item Doing 19 Experimental System Z to A Doing Known-Item Learning 20 Baseline System Known-Item Learning Doing 21 Experimental System A to Z Learning Doing Known-Item 22 Baseline System Doing Learning Known-Item 23 Experimental System Z to A Known-Item Doing Learning *Participant 1 and 2 were pilots of the user study, hence are not included in the analysis **Participant 3 was discarded. Systems can be accessed here: - Baseline System: http://diigubc.ca/cd_study_1 - Experimental System A to Z: http://diigubc.ca/cd_study_2 - Experimental System Z to A: http://diigubc.ca/cd_study_3 (Please note that these are experimental systems which might be accessible at all times.) Table A40: Study 2 - System and Activity Assignment 115 Appendix H ? Study 2: Scenarios Doing: An elderly uncle has had a stroke and is confined to a wheelchair, but he and your aunt want to continue to live in their own home. You are seeking information on how adapt their home to the new circumstances. Known-Item: You are performing historical research into First Nations communities and are looking for records of individuals. You have heard that it is possible to obtain these records from a federal government agency. You are looking for the official document needed to send an information request to this agency. Learning: After listening to an interesting radio program about weather disasters, you want to learn more about the effects of extreme weather situations and their impact on different communities in Canada. You are seeking information to learn about this topic. 116 Appendix I ? Study 2: Protocol of Activities in Experimental User Study A. Greet the participant B. Provide the participant with the consent form and allow sufficient time for the participant to completely read it, collect signed copy and give second copy to participant C. Answer any questions the participant might have D. Present the participant with the pre-questionnaire (Background Information Questionnaire) and allow sufficient time for the participant to complete it E. Introduce FRED system to participant, and stress that only this system can be used Introduction: The FRED system allows for searching a small sub set of Government Canada web content. It allows for simple text search. (And in case of the second system instance: It offers filter categories to refine the search query.) F. Answer any questions the participant might have G. Explain the parts of the search scenario activities to the participant and clarify that it the search is not a test about finding the right documents. Explain that the actions on the screen will be recorded, but no video and audio recording will take place. Also ask if it is ok to observe and take notes. H. Present the participant with task sheet and pre-task questions, and allow sufficient time for the participant to completely read both, and respond to any questions the participant might have I. Allow sufficient time for the participant to perform the task, but no more than 10 minutes J. Once the participant indicates that the task has been completed or cannot be completed, present the post-task questions K. Note the type of task (doing, learning, or known-item) and the system instance (1,2, or 3) on the questionnaire L. Repeat H through K for each additional task presented to the participant M. After the post-task questions of the last task have been answered, present Part A of the post-questionnaire to the participant N. Once Part A has be answered by the participant announce that you will ask a few additional questions and record the participant?s answers, turn on audio recording device and ask questions to participant O. After the final set of questions has been answered by the participant, thank the participant, answer any remaining questions, hand over the honorarium and ask the participant to sign the receipt form. Important: Do not forget to write participant number on all documents EXCEPT consent form and receipt form. 117 Appendix J ? Study 2: Pre-Questionnaire In terms of searching the Internet how do you rate your skill level on a scale from 1 to 7? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? What is your age? ? 21 or younger ? 22 to 31 ? 32 to 41 ? 42 to 51 ? 52 to 61 ? 62 or older ? Prefer not to answer What is your gender: ? Female ? Male ? Other ? Prefer not to tell Are you currently a student? ? Yes, studying full time ? Yes, studying part time ? No, I am not currently undertaking formal study Are you currently employed? ? Yes, working full time ? Yes, working part time ? No, I am not currently employed. What is the highest degree or level of school you have completed? If currently enrolled, mark the program or degree that is in progress. ? Public or high school, no diploma ? High school diploma or equivalent ? Apprenticeship or trades certificate or diploma ? College, CEGEP or other non-university certificate or diploma ? Bachelor's degree ? Degree in medicine, dentistry, veterinary medicine or optometry ? Master's degree ? Earned doctorate ? Other (please specify) _________________________________ What is your status in Canada: ? Canadian Citizen ? Permanent Resident of Canada ? Other (please specify) ____________________________________ 118 Appendix K ? in Study 2: Search Instructions and Questionnaires Search Task Part A Please read the scenario on the card and rank these questions on a scale from 1 to 7. 1. How realistic is the scenario as a situation in which you might search for information? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? 2. How would you rate your personal level of knowledge about the topic of this scenario? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? Part B Search the FRED system for information that would help you with this scenario. Bookmark any useful pages you find and stop when you have searched enough. You have about 10 minutes. 119 Part C Answer the next set of questions when you have finished searching. 1. How satisfied are you with the outcome of your search? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? 2. How challenging was it to carry out this search task? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? 3. To what extent do you think that you found what is needed to complete the task? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? 4. Having completed this search, how do you rate your personal level of knowledge about the topic of this scenario? 1 (Low) 2 3 4 5 6 7 (High) ? ? ? ? ? ? ? 5. If you are not fully satisfied with the outcome of the search: a. What kind of information would you have liked to find? ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ b. What kind of search system features do you think would have helped? ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 120 Appendix L ? Study 2: Post-Questionnaire PART A ? To be completed by participant 1. On a scale from 1 to 7, please indicate your level of agreement with the following statements. 1 (Low) 2 3 4 5 6 7 (High) I found the FRED system to be useful in completing the tasks. ? ? ? ? ? ? ? My interaction with the FRED system was clear and understandable. ? ? ? ? ? ? ? Interacting with the FRED system does not require a lot of mental effort on my part. ? ? ? ? ? ? ? I found the FRED system to be easy to use. ? ? ? ? ? ? ? 2. On a scale from 1 to 7, how do you rate the usefulness of the following search filters in finding information to complete the tasks?62 1 (Low) 2 3 4 5 6 7 (High) Did not use Audience ? ? ? ? ? ? ? ? Date Published ? ? ? ? ? ? ? ? Department ? ? ? ? ? ? ? ? Length (of document) ? ? ? ? ? ? ? ? Location ? ? ? ? ? ? ? ? Type (of document) ? ? ? ? ? ? ? ? 62 This question was only presented to participants using the experimental system. 121 PART B ? To be asked by interviewer 1. What parts of the FRED system did you find useful in finding information to complete the tasks? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 2. What system features not available in the FRED system would have been useful in finding information to complete the tasks? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ (optional) 3. During task x, I noticed that you y. Can you explain what was going on there? What were you trying to do? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 122 Appendix M ? Study 2: Normality and Variance Details Table A41 Study 2 - tests of Normality for Satisfaction Measures Measure Facet Availability Kolmogorov-Smirnova Shapiro-Wilk Normally Distributed Statistic df Sig. Statistic df Sig. Perceived Ease Of Use No Facets .157 10 .200* .929 10 .436 Yes With Facets .210 10 .200* .906 10 .256 Yes Level of Satisfaction No Facets .193 30 .006 .900 30 .008 No With Facets .188 30 .008 .887 30 .004 No Level of Challenge No Facets .164 30 .038 .922 30 .031 No With Facets .246 30 .000 .902 30 .010 No *. This is a lower bound of the true significance. a. Lilliefors Significance Correction Table A41: Study 2 - tests of Normality for Satisfaction Measures Table A42 Study 2 - tests of Normality for Effectiveness Measures Measure Facet Availability Kolmogorov-Smirnova Shapiro-Wilk Normally Distributed Statistic df Sig. Statistic df Sig. Perceived Success No Facets .213 30 .001 .889 30 .005 No With Facets .147 30 .095 .892 30 .005 Yes Perceived Knowledge Gain No Facets .284 30 .000 .878 30 .002 No With Facets .191 30 .007 .947 30 .144 Yes Number of Documents Bookmarked Total No Facets .206 30 .002 .810 30 .000 No With Facets .196 30 .005 .662 30 .000 No Number of Documents Bookmarked FRED Accessible No Facets .320 30 .000 .695 30 .000 No With Facets .276 30 .000 .737 30 .000 No Relevance Assessment Manual All URLs No Facets .098 30 .200* .933 30 .058 Yes With Facets .103 28 .200* .951 28 .208 Yes Relevance Assessment Participants All URLs No Facets .163 30 .042 .820 30 .000 No With Facets .182 28 .018 .782 28 .000 No Relevance Assessment Manual FRED Accessible URLs No Facets .195 22 .029 .875 22 .010 Yes With Facets .268 20 .001 .865 20 .010 Yes Relevance Assessment Participants FRED Accessible URLs No Facets .119 22 .200* .961 22 .502 Yes With Facets .210 20 .022 .809 20 .001 No a. Lilliefors Significance Correction Table A42: Study 2 - tests of Normality for Effectiveness Measures 123 Table A43 Study 2 - tests of Normality for Efficiency Measures Measure b,c,e,f,g,h Facet Availability Kolmogorov-Smirnova Shapiro-Wilk Normally Distributed Statistic df Sig. Statistic df Sig. TAM Perceived Usefulness No Facets .277 10 .028 .892 10 .177 Yes With Facets .155 10 .200* .969 10 .886 Yes Completion Time (s) No Facets .173 30 .023 .893 30 .006 No With Facets .136 29 .178 .952 29 .204 Yes Number of Text Queries No Facets .207 30 .002 .655 30 .000 No With Facets .245 29 .000 .764 29 .000 No Number of FRED Results Lists Viewed No Facets .229 30 .000 .809 30 .000 No With Facets .151 29 .088 .909 29 .016 No Number of Facet Interactions With Facets .148 29 .105 .907 29 .015 No Number of Facet Filter Interactions With Facets .215 29 .001 .844 29 .001 No Number of Total Documents Viewed No Facets .150 30 .082 .919 30 .026 No With Facets .159 29 .057 .879 29 .003 No Number of FRED Accessible Documents Viewed No Facets .196 30 .005 .880 30 .003 No With Facets .226 29 .001 .877 29 .003 No Number of Total Documents Viewed Per Minute No Facets .082 30 .200* .964 30 .380 Yes With Facets .141 29 .147 .889 29 .005 No Number of FRED Accessible Documents Viewed Per Minute No Facets .140 30 .138 .932 30 .056 Yes With Facets .186 29 .012 .848 29 .001 No Number of Total Documents Viewed Per Text Query No Facets .178 30 .016 .851 30 .001 No With Facets .319 29 .000 .474 29 .000 No Number of FRED Accessible Documents Viewed Per Text Query No Facets .180 30 .015 .852 30 .001 No With Facets .264 29 .000 .649 29 .000 No Number of Total Documents Viewed Per Results List Viewed No Facets .235 30 .000 .770 30 .000 No With Facets .417 29 .000 .309 29 .000 No Number of FRED Accessible Documents Viewed Per Results List Viewed No Facets .253 30 .000 .798 30 .000 No With Facets .257 29 .000 .593 29 .000 No Number of Total Documents Viewed Per Facet Interaction With Facets .206 29 .003 .799 29 .000 No Number of FRED Accessible Documents Viewed Per Facet Interaction With Facets .259 29 .000 .750 29 .000 No Number of Total Documents Viewed Per Facet Filter Interaction With Facets .255 29 .000 .628 29 .000 No Number of FRED Accessible Documents Viewed Per Facet Filter Interaction With Facets .271 29 .000 .768 29 .000 No *. This is a lower bound of the true significance. a. Lilliefors Significance Correction b. Number of Facet Interactions is constant when Facet Availability = No Facets. It has been omitted. c. Number of Facet Filter Interactions is constant when Facet Availability = No Facets. It has been omitted. e. Number of Total Documents Viewed Per Facet Interaction is constant when Facet Availability = No Facets. It has been omitted. f. Number of FRED Accessible Documents Viewed Per Facet Interaction is constant when Facet Availability = No Facets. It has been omitted. g. Number of Total Documents Viewed Per Facet Filter Interaction is constant when Facet Availability = No Facets. It has been omitted. h. Number of FRED Accessible Documents Viewed Per Facet Filter Interaction is constant when Facet Availability = No Facets. It has been omitted. Table A43: Study 2 - tests of Normality for Efficiency Measures 124 Table A44 Study 2 - Independent-Samples Mann-Whitney U test Statistics for Satisfaction Measures with Grouping Variable Facet Availability Measure Total N Mann-Whitney U Wilcoxon W test Statistic Std. Error Standardized test Statistic Asym. Sig. (2-sided) Perceived Ease of Use 20 43.500 98.500 43.500 13.139 -.495 .6211 Level of Satisfaction 60 418.500 883.500 418.500 66.273 -.475 .635 Level of Challenge 60 443.500 908.500 443.500 66.702 -.097 .922 Asymptotic significances are displayed. The significance level is .05. 1 Exact significance for this test is .631 Table A44: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Satisfaction Measures with Grouping Variable Facet Availability Table A45 Study 2 - Independent-Samples Mann-Whitney U test Statistics for Effectiveness Measures with Grouping Variable Facet Availability Measure Total N Mann-Whitney U Wilcoxon W test Statistic Std. Error Standardized test Statistic Asym. Sig. (2-sided) Perceived Success 60 450.500 915.500 450.500 66.392 .008 .994 Perceived Knowledge Gain 60 459.000 924.000 459.000 65.182 .138 .890 Number of Documents Bookmarked Total 60 427.000 892.000 427.000 67.043 -.343 .732 Number of Documents Bookmarked FRED Accessible 60 421.000 886.000 421.000 65.721 -.441 .659 Relevance Assessment Manual All URLs 60 441.000 847.000 441.000 63.997 .328 .743 Relevance Assessment Participants All URLs 58 360.000 766.000 360.000 64.217 -.934 .350 Relevance Assessment Manual FRED Accessible URLs 42 229.500 439.500 229.500 38.492 .247 .805 Relevance Assessment Participants FRED Accessible URLs 42 166.000 376.000 166.000 39.612 -1.363 .173 Asymptotic significances are displayed. The significance level is .05. Table A45: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Effectiveness Measures with Grouping Variable Facet Availability 125 Table A46 Study 2 - Independent-Samples Mann-Whitney U test Statistics for Efficiency Measures with Grouping Variable Facet Availability Measure Total N Mann-Whitney U Wilcoxon W test Statistic Std. Error Standardized test Statistic Asym. Sig. (2-sided) Perceived Usefulness 20 44.000 99.000 44.000 12.998 -.462 .6441 Completion Time (s) 60 399.000 864.000 399.000 67.638 -.754 .451 Number of Text Queries 60 375.000 840.000 375.000 67.075 -1.118 .264 Number of FRED Results Lists Viewed 60 541.500 1,006.500 541.500 67.402 1.358 .175 Number of Facet Interactions 60 825.000 1,290.000 825.000 60.538 6.194 .000 Number of Facet Filter Interactions 60 810.000 1,275.000 810.000 59.862 6.014 .000 Number of Total Documents Viewed 60 358.500 823.500 358.500 67.493 -1.356 .175 Number of FRED Accessible Documents Viewed 60 336.500 801.500 336.500 67.092 -1.692 .091 Number of Total Documents Viewed Per Minute 60 385.000 850.000 385.000 67.639 -.961 .337 Number of FRED Accessible Documents Viewed Per Minute 60 384.000 849.000 384.000 67.525 -.977 .328 Number of Total Documents Viewed Per Text Query 60 437.000 902.000 437.000 67.563 -.192 .847 Number of FRED Accessible Documents Viewed Per Text Query 60 372.500 837.500 372.500 67.396 -1.150 .250 Number of Total Documents Viewed Per Results List Viewed 60 316.000 781.000 316.000 67.614 -1.982 .047 Number of FRED Accessible Documents Viewed Per Results List Viewed 60 285.000 750.000 285.000 67.457 -2.446 .014 Number of Total Documents Viewed Per Facet Interaction 60 825.000 1,290.000 825.000 60.555 6.193 .000 Number of FRED Accessible Documents Viewed Per Facet Interaction 60 765.000 1,230.000 765.000 57.609 5.468 .000 Number of Total Documents Viewed Per Facet Filter Interaction 59 795.000 1,230.000 795.000 58.670 6.136 .000 Number of FRED Accessible Documents Viewed Per Facet Filter Interaction 60 735.000 1,170.000 735.000 55.611 5.395 .000 Asymptotic significances are displayed. The significance level is .05. 1 Exact significance for this test is .684 Table A46: Study 2 - Independent-Samples Mann-Whitney U test Statistics for Efficiency Measures with Grouping Variable Facet Availability 126 Appendix N ? Study 2: Sequence Variance Details Table A47 Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence Measure Total N test Statistic df Asymp. Significance (2-sided) Completion Time (s) 60 2.004 2 .367 Number of Text Queries 60 1.017 2 .601 Number of Total Documents Viewed 60 .030 2 .985 Number of FRED Accessible Documents Viewed 60 .625 2 .732 Number of Facet Interactions 60 1.788 2 .409 Number of Facet Filter Interactions 60 1.381 2 .501 Number of FRED Results Lists Viewed 60 3.003 2 .223 Number of Total Documents Viewed Per Minute 60 .739 2 .691 Number of FRED Accessible Documents Viewed Per Minute 60 .786 2 .675 Number of Total Documents Viewed Per Text Query 60 .445 2 .801 Number of FRED Accessible Documents Viewed Per Text Query 60 .607 2 .738 Number of Total Documents Viewed Per Results List Viewed 60 .531 2 .767 Number of FRED Accessible Documents Viewed Per Results List Viewed 60 .518 2 .772 Number of Total Documents Viewed Per Facet Interaction 60 .052 2 .974 Number of FRED Accessible Documents Viewed Per Facet Interaction 60 .736 2 .692 Number of Total Documents Viewed Per Facet Filter Interaction 59 .295 2 .863 Number of FRED Accessible Documents Viewed Per Facet Filter Interaction 59 .312 2 .856 Relevance Assessment Manual All URLs 58 3.035 2 .219 Relevance Assessment Participants All URLs 58 2.053 2 .358 Relevance Assessment Manual FRED Accessible URLs 42 5.537 2 .063 Relevance Assessment Participants FRED Accessible URLs 42 .408 2 .815 Perceived Success 60 8.673 2 .013 Perceived Knowledge Gain 60 2.206 2 .332 Number of Total Documents Bookmarked 60 .846 2 .655 Number of FRED Accessible Documents Bookmarked 60 .020 2 .990 Level of Satisfaction 60 9.379 2 .009 Level of Challenge 60 5.517 2 .063 Asymptotic significances are displayed. The significance level is .05. Perceived Usefulness and Perceived Ease of Use are not included in this analysis as they were not collected for each scenario, but only in the post-questionnaire independently from the scenarios. Table A47: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence Table A48 Study 2 - Mann Whitney U test for Pairwise Comparisons of All Measures Between Sequence of Task Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Perceived Success 1st - 2nd -11.275 5.421 -2.080 .038 .113 1st ? 3rd -15.425 5.421 -2.845 .004 .013 2nd ? 3rd -4.150 5.421 -.766 .444 1.000 Level of Satisfaction 1st - 2nd -11.525 5.411 -2.130 .033 .100 1st ? 3rd -16.075 5.411 -2.971 .003 .009 2nd ? 3rd -4.550 5.411 -.841 .400 1.000 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A48: Study 2 - Mann Whitney U test for Pairwise Comparisons of All Measures Between Types of Tasks 127 Table A49 Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Baseline System Measure Total N test Statistic df Asymp. Significance (2-sided) Completion Time (s) 30 .279 2 .870 Number of Text Queries 30 .218 2 .897 Number of Total Documents Viewed 30 1.457 2 .483 Number of FRED Accessible Documents Viewed 30 4.270 2 .118 Number of FRED Results Lists Viewed 30 .404 2 .817 Number of Total Documents Viewed Per Minute 30 2.108 2 .348 Number of FRED Accessible Documents Viewed Per Minute 30 6.834 2 .033 Number of Total Documents Viewed Per Text Query 30 3.858 2 .145 Number of FRED Accessible Documents Viewed Per Text Query 30 3.999 2 .135 Number of Total Documents Viewed Per Results List Viewed 30 3.598 2 .165 Number of FRED Accessible Documents Viewed Per Results List Viewed 30 4.423 2 .110 Relevance Assessment Manual All URLs 30 .588 2 .745 Relevance Assessment Participants All URLs 30 2.277 2 .320 Relevance Assessment Manual FRED Accessible URLs 22 4.001 2 .135 Relevance Assessment Participants FRED Accessible URLs 22 1.072 2 .585 Perceived Success 30 1.544 2 .462 Perceived Knowledge Gain 30 .710 2 .701 Number of Total Documents Bookmarked 30 6.028 2 .049 Number of FRED Accessible Documents Bookmarked 30 7.605 2 .022 Level of Satisfaction 30 .423 2 .423 Level of Challenge 30 1.089 2 .580 Asymptotic significances are displayed. The significance level is .05. Perceived Usefulness and Perceived Ease of Use are not included in this analysis as they were not collected for each scenario, but only in the post-questionnaire independently from the scenarios. Table A49: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Baseline System 128 Table A50 Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Baseline System Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Number of FRED Accessible Documents Viewed Per Minute 1st - 2nd -4.450 3.932 -1.132 .258 .773 1st ? 3rd 5.800 3.932 1.475 .140 .421 2nd ? 3rd 10.250 3.932 2.607 .009 .027 Number of Total Documents Bookmarked 1st - 2nd 2.700 3.897 .693 .488 1.000 1st ? 3rd 9.300 3.897 2.386 .017 .051 2nd ? 3rd 6.600 3.897 1.693 .090 .271 Number of FRED Accessible Documents Bookmarked 1st - 2nd -1.900 3.831 -.496 .620 1.000 1st ? 3rd 8.050 3.831 2.101 .036 .107 2nd ? 3rd 9.950 3.831 2.597 .009 .028 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A50: Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Baseline System Table A51 Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Experimental System Measure Total N test Statistic df Asymp. Significance (2-sided) Completion Time (s) 30 2.955 2 .228 Number of Text Queries 30 2.467 2 .291 Number of Total Documents Viewed 30 2.456 2 .293 Number of FRED Accessible Documents Viewed 30 10.677 2 .005 Number of Facet Interactions 30 7.155 2 .028 Number of Facet Filter Interactions 30 3.339 2 .188 Number of FRED Results Lists Viewed 30 3.970 2 .137 Number of Total Documents Viewed Per Minute 30 5.546 2 .062 Number of FRED Accessible Documents Viewed Per Minute 30 11.154 2 .004 Number of Total Documents Viewed Per Text Query 30 2.767 2 .251 Number of FRED Accessible Documents Viewed Per Text Query 30 10.657 2 .005 Number of Total Documents Viewed Per Results List Viewed 30 4.428 2 .109 Number of FRED Accessible Documents Viewed Per Results List Viewed 30 11.499 2 .003 Number of Total Documents Viewed Per Facet Interaction 30 1.343 2 .511 Number of FRED Accessible Documents Viewed Per Facet Interaction 30 2.963 2 .227 Number of Total Documents Viewed Per Facet Filter Interaction 29 .102 2 .950 Number of FRED Accessible Documents Viewed Per Facet Filter Interaction 29 1.150 2 .563 Relevance Assessment Manual All URLs 28 4.170 2 .124 Relevance Assessment Participants All URLs 28 1.105 2 .576 Relevance Assessment Manual FRED Accessible URLs 20 1.863 2 .394 Relevance Assessment Participants FRED Accessible URLs 20 3.713 2 .156 Perceived Success 30 8.208 2 .017 Perceived Knowledge Gain 30 1.539 2 .463 Number of Total Documents Bookmarked 30 1.296 2 .523 Number of FRED Accessible Documents Bookmarked 30 8.363 2 .015 Level of Satisfaction 30 9.018 2 .011 Level of Challenge 30 4.906 2 .086 Asymptotic significances are displayed. The significance level is .05. Perceived Usefulness and Perceived Ease of Use are not included in this analysis as they were not collected for each scenario, but only in the post-questionnaire independently from the scenarios. Table A51: Study 2 - Independent-Samples Kruskal-Wallis test Statistics for All Measures with Grouping Variable Sequence limited to Experimental System 129 Table A52 Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Experimental System Sample1 ? Sample2 test Statistic Std. Error Std. test Statistic Sig. Adj. Sign. Number of FRED Accessible Documents Viewed 1st - 2nd 5.500 3.882 1.417 .157 .470 1st ? 3rd -7.150 3.882 -1.842 .066 .197 2nd ? 3rd -12.650 3.882 -3.258 .001 .003 Number of Facet Interactions 1st - 2nd 7.200 3.919 1.837 .066 .199 1st ? 3rd 10.200 3.919 2.602 .009 .028 2nd ? 3rd 3.000 3.919 .765 .444 1.000 Number of FRED Accessible Documents Viewed Per Minute 1st - 2nd 3.500 3.928 .891 .373 1.000 1st ? 3rd -9.200 3.928 -2.342 .019 .058 2nd ? 3rd -12.700 3.928 -3.233 .001 .004 Number of FRED Accessible Documents Viewed Per Text Query 1st - 2nd 2.900 3.922 .739 .460 1.000 1st ? 3rd -9.350 3.922 -2.384 .017 .051 2nd ? 3rd -12.250 3.922 -3.123 .002 .005 Number of FRED Accessible Documents Viewed Per Results List Viewed 1st - 2nd 2.900 3.925 .739 .460 1.000 1st ? 3rd -9.800 3.925 -2.497 .013 .038 2nd ? 3rd -12.700 3.925 -3.236 .001 .004 Perceived Success 1st - 2nd -8.250 3.859 -2.138 .033 .098 1st ? 3rd -10.500 3.859 -2.721 .007 .020 2nd ? 3rd -2.250 3.859 -.583 .560 1.000 Number of FRED Accessible Documents Bookmarked 1st - 2nd 2.150 3.816 .563 .573 1.000 1st ? 3rd -8.300 3.816 -2.175 .030 .089 2nd ? 3rd -10.450 3.816 -2.738 .006 .019 Level of Satisfaction 1st - 2nd -8.700 3.852 -2.259 .024 .072 1st ? 3rd -10.950 3.852 -2.843 .004 .013 2nd ? 3rd -2.250 3.852 -.584 .559 1.000 Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same. Asymptotic significances (2-sided tests) are displayed. The significance level is .05. Table A52: Study 2 - Mann Whitney U test for Pairwise Comparisons for All Measures Between Sequence of Task limited to Experimental System "@en ; edm:hasType "Thesis/Dissertation"@en ; vivo:dateIssued "2013-11"@en ; edm:isShownAt "10.14288/1.0074206"@en ; dcterms:language "eng"@en ; ns0:degreeDiscipline "Library and Information Studies"@en ; edm:provider "Vancouver : University of British Columbia Library"@en ; dcterms:publisher "University of British Columbia"@en ; dcterms:rights "Attribution-NonCommercial-NoDerivatives 4.0 International"@en ; ns0:rightsURI "http://creativecommons.org/licenses/by-nc-nd/4.0/"@en ; ns0:scholarLevel "Graduate"@en ; dcterms:title "Task-type-specific use of facets in discovering online content"@en ; dcterms:type "Text"@en ; ns0:identifierURI "http://hdl.handle.net/2429/44960"@en .
https://open.library.ubc.ca/media/download/rdf-turtle/24/1.0074206/0.txt
open.library.ubc.ca
crawl=CC-MAIN-2019-39
322543
application/http; msgtype=response
sha1:KD3QPNFWEUM4IFCCVMB6MKKJ4OGN6VGP
<urn:uuid:f94a08e0-18c3-4401-9639-f214ad39eb8c>
2019-09-23T01:23:51Z
142.103.96.89
text/plain
sha1:TXCJ2Z5ZNWDATX6SJCNEFC725GHRT3TW
<urn:uuid:dfe8fa51-81c6-44ff-bc7b-826dfccde79d>
https://open.library.ubc.ca/media/download/rdf-turtle/24/1.0074206/0.txt
response
<urn:uuid:b32e75ca-a2ab-4850-b93f-45e2dc6c0275>
isPartOf: CC-MAIN-2019-39
publisher: Common Crawl
description: Wide crawl of the web for September 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-102.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/