A Trustable Recommender System for Web Content

A Trustable Recommender System for Web Content
of 6
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  A Trustable Recommender System for Web Content Tingshao Zhu, Russ Greiner Dept. of Computing ScienceUniversity of AlbertaCanada T6G 2E8 Gerald H ¨aubl School of BusinessUniversity of AlbertaCanada T6G 2R6 Bob Price, Kevin Jewell Dept. of Computing ScienceUniversity of AlbertaCanada T6G 2E8 1 ABSTRACT This paper presents three principles for building a Web rec-ommender system that can be trusted to provide valuablepage recommendations to Web users. We also demon-strate the implementation of these principles in an effectivecomplete-Web recommender system —  WebICLite .  Web- ICLite  is able to predict relevant pages based on a learnedmodel, whose parameters are estimated from a labelled cor-pus. Data from a recent user study demonstrate that the pre-diction model can recommend previously unseen pages of high relevance from anywhere on the Web for any user. 1.1 Keywords Browsing Behavior Model, Machine Learning, InformationSearch, Web Content Recommendation 2 Introduction While the World Wide Web contains a vast quantity of in-formation, it is often time consuming and difficult for webusers To find the information they are seeking on the Web.It is natural to ask whether computational techniques can beused to assist the user in finding useful pages. Today, mil-lions of users employ information retrieval techniques in theform of popular search engines to successfully find usefulpages.In order to benefit from these search engines, users must ex-plicitly perform searches, and thus they must have intuitionsabout what keywords they should use to efficiently circum-scribe the information they are seeking within the billions of Web pages that search engines typically index. We believe,however, that the time has come for information systems togo beyond simple fulfillment of specific requests. We en-vision interfaces that actively make suggestions for usefulcontent, i.e., Web recommender systems.We envision Web recommender systems that generate under-standable and trustable recommendations, and present theserecommendations to users in a natural way, and even includeinterfaces can encourage users to provide ratings in order to“close the loop” for recommendations. For example, it isimportant for the system to help users understand the itemsrecommended and then evaluate the recommendations.To build a Web recommender system, it is important to un-derstand both what recommender systems are, and how theycan be applied effectively. In this paper, we introduce threeprinciples that we followed as we built our trustable Webrecommender system —  WebICLite : Useful and Complete-Web Recommendation:  To gainthe trust from Web users, it is critical to generate usefulrecommendations, as less relevant recommendationswill discourage users. Since the Internet contains a vastamount of information spread across multiple sites, weobserved that to satisfy the current information need,it is important to provide the most relevant pages from anywhere on the Web . 1 Adaptive:  A user’s needs can change dramatically even inthe course of a single browsing session. As the userworks on various tasks and subtasks, s/he will often re-quire information on various unrelated topics, includingtopics the user has investigated before. This required usto build a system that can suggest recommendations dy-namically, that is, the system needs to be adaptive to thecurrent user and browsing session. User interface:  In addition to generating useful recommen-dations, it is also critically important to design an effi-cient mechanism for presenting these recommendationsto users — one that is compatible with a user’s naturalbehavior while using the Web. If the recommender in-terface is different from traditional interfaces, then peo-ple will have to change their behavior in order to adaptto the new interface.One of our research goals is to understand how best topresent recommendations to users so as to encouragethem to explicitly rate content recommendations (e.g.,how good is this recommendation?) Such relevancefeedback can subsequently be used to tune the perfor-mance of the recommender system. 1 We use the phrase “anywhere in the Web” to mean “anywhere thatGoogle has indexed”. 1  We propose that there exists a general user-independentbrowsing behavior model of goal-directed informationsearch on the web. That is, people follow some very gen-eral rules to locate the information they are seeking. If we(or our algorithms) can detect such patterns, then our sys-tems can provide more useful content recommendations tothe user. In our research, we use machine learning to ac-quire browsing behavior models, which can then be used togenerate useful recommendations; we also consider learning  personalized   models.There are many advantages to learning a model from data —i.e., to tuning the model’s internal parameters based on train-ing examples First, a system that adapts to the user’s actualactions will perform better than one that does not. Of course,we could have a person inspect the earlier performance, andthen make these modifications, by hand. A machine learningalgorithm, however, can discover features in datasets that arebeyond the scope of human inspection, as they too large ortoo complex.We developed the  WebICLite  Web recommender system tobe “passive”, in that it can recommend pages relevant to theuser’s  current   information need without requiring the userto do any additional work (e.g., the user does not need toanswerintrusivequestions, etc.). Likemostrecommendationsystems,  WebICLite  watches a user as she navigates througha sequence of pages, and suggests pages that (it predicts) willprovide the relevant information. WebICLite  is unique in several respects. First, while manyrecommendation systems are server-side and hence specificto a single web site [9, 10], our client-side system is not sospecific, and so can point users to pages  anywhere on theWeb . Secondly,  WebICLite  can predict the user’s informationneed  dynamically , based on his/her current context (e.g., thecurrent session). The third difference deals with the goalof the recommendation system: our goal is to recommendonly useful pages; i.e., pages that are relevant to the user’stask. These “Information Content” pages (aka IC-pages) are just the pages the user needs to solve his/her current task,and not the overhead pages required to reach them. Thisdiffers from systems that instead attempt to lead the user topages that similar users have visited earlier (independent of whether those “familiar” pages in fact contain the answer tothe user’s current quest).Section 3 discusses related work. Section 4 then describe asimple procedure for training the model, and the results of a user study (i.e., LILAC) that demonstrates the ability of our model to predict pages useful to the current user, fromanywhere in the Web.Section 5 describes an implementationof our ideas in the form of a stand-alone web browser,  Web- ICLite , that runs on theuser’s computer, and provideson-linerecommendations, to pages anywhere on the Web (not juston the user’s current Web site). Finally, Section 6 concludeswith a summary of our key contributions and insights. 3 Related Work Many groups have built various types of systems that rec-ommend pages to web users. This section will summarizeseveral of those systems, and discuss how they differ fromour approach.Zukerman [15] distinguishes two main approaches to learn-ing which pages will appeal to a user: A  content-based   sys-tem tries to learn a model based on the contents of the Webpage, while a  collaborative  system bases its model on find-ing “similar” users, and assuming the current user will likethe pages that those similar users have visited.Our  WebICLite  system is basically content-based, as its de-cisions are based on the combination of the page contentand the applied actions. However, our approach differs fromthe  standard   content-based systems: As many such systemsare restricted to a single Web site, their classifiers can bebased on a limited range of words or URLs; this meansthey can make predictions about the importance of specificURLs (see association rules [1]), or of specific hard-selectedwords [4, 7, 3]. We did not want to restrict ourselves to asingle Web site, but wanted a system that could recommendpages anywhere on the Web; this meant we did not want torestricted the range of words, or pages. For this reason, webuilt our content-based classifiers based on characteristics(“browsing properties”) of the words. As a result, our sys-tem is not restricted to predefined words, nor to Web pagesthat already have been visited by this user, nor even to Webpages that have been visited by similar users. As we expectthat these  browsing properties  will be common accross webusers, we expect them to be useful even for novel users, andso help them find pages even in novel web environments.Another way to predict useful Web pages is to take advan-tage of a Web user model. Chi et al. [6], and Pirolli andFu [11] construct Information Need from the context of thefollowed hyperlinks based on the theory of “InformationScent”, and view it as the information that the user wants.While this appears very similar to our approach, note thatour approach is based on the idea that some browsing prop-erties of a word (e.g., context around hyperlinks, or wordappearing in titles, etc.) indicate whether a user considers aword to be important  in this current context  ; moreover, oursystem has  learned   this “which words are important” predi-catefromtrainingdata, ratherthanbyhand-codinganad-hocassumption. Letizia [8] and Watson [5] anticipate the user’sinformation needs using heuristics. While the heuristics mayrepresent the user behavior well, we believe a model learnedfrom user data, should produce a more accurate user model.Table 1 compares the following types of recommender sys-tems in terms of their key properties: •  COB: Co-occurrence Based e.g., Association Rule [1],Sequential Pattern [2], etc. •  CF: Collaborative Filtering [12] •  CB: Content-Based [4, 7, 3] •  HBM: Heuristic-Based Model [11, 8, 5] •  IC-Models: IC-based Models; see Section 4.2  COB CF CB HBM IC-ModelsSpecific Site/Domain Yes Yes Yes No NoModel Acquisition Learning Learning Learning Hand-coded LearningAnnotation Required (training) No No Yes No YesAnnotation Required (performance) No No No No NoReference Set Population Group Individual None Individual/Group/PopulationRecommending Useful Information No N/A Yes Yes YesUsing Sequential Information No No No Yes YesTable 1: Techniques for Recommender SystemFigure 1: Browsing Behavior to Locate IC-pageThe most common kinds of collaborative filtering are basedon weakly informative choice signals from users (purchasesatAmazon, viewsofawebpage, etc.). Weviewthesesystemas not requiring user annotation. In some systems like moviedatabases, users explicitly rate the value of items. We wouldtreat the explicit recommendations as annotations. 4 Learning User-Web Interaction Models Our work on web browsing behavior models is driven byseveral observations we have made concerning the nature of human-computer interaction while searching information onthe Web.Consider the example suggested by Figure 1. Imagine theuser needs information about marine animals. The user seesa page with links on “Dolphins” and “Whales”. Clicking onthe “Dolphins” link takes the user to a page about the NFLFootball team. This page is not an IC-page for this user atthis time, so the user “backs up”. We might conclude thatthe word “Football”, which appears on the previous page butnot on the current page, might not be an IC-word. The userthen tries the “Whale” pointer, which links to a page whosetitle includes “whale”, and which includes whales, oceans,and other marine terms in its content. The user then followsa link on that page to another with similar content, and soforth, until terminating on a page about a local aquarium. Wemight conclude that words such as “Whale” and “Ocean” areIC-words, but “football” is not.The above observation suggest that the user’s current infor-mation need can be identified from the pages the user visitsand the actions that the user applies to the pages. The contentof the page is communicated by the roles that words play onthe page. We make the simplifying assumption that we canrepresent the information need of the session by a set of sig-nificant words from the session. We further assume that thesignificance of words can be judged independently for eachword from the roles played by instances of the word on pagesthroughout the sequence (e.g., appearing as plain text, high-lighted in a title, appearing in followed hyperlink, etc.). Tocapture the reaction of the Web user, we developed a numberof browsing features from the page sequence. The completelist of these ”browsing features” can be found in our previouspapers [13, 14].To obtain the user model for Web recommendation, we firstcollected a set of annotated web logs (where the user hasindicated which pages are IC-pages), from which our learn-ing algorithm learned to characterize the IC-page associatedwith the pages in any partial subsession. Again, see [13, 14]for the details. 4.1 LILAC Study The five-week study, named “LILAC” (Learn from the In-ternet: Log, Annotation, Content), attempts to evaluate thebrowsing behavior models that we proposed. In particular,it is designed to qualitatively assess the strengths of differ-ent browsing behavior based models, relative to a baselinemodel, and to gather data for further studies. We want toevaluate these models on actual users working on day-to-daytasks, to determine if the models help users, currently vis-iting arbitrary pages, find useful information from arbitrarysites on the Web.The process begins when the participants install the  WebIC  software 2 on their own computer, and begin browsing theirown choice of web pages. 3 Whenever the subject discov-ers an “Information Content” page, s/he is instructed to labelthis page as an IC-page. Here,  WebIC   will ask the subjectto compare the current IC-page to an alternative page that it 2 As the name suggests, this  WebIC   system is the srcinal system, whichhas evolved into the current  WebICLite . 3 We requested they use English language pages, but of course, did notenforce this. We also gave them ways to turn off the observation part of  WebIC   when dealing with private information — e.g., email or banking. 3  provides. The subject can choose one of the following op-tions: •  Fully answered my question •  Relevant, but does not answer my question fully •  Interesting, but not so relevant •  Remotely related, but still in left field •  Not related at allThe only other interaction is when the subjects could not finda page that addresses his/her information need, and so clickson the “Suggest” button. Here,  WebIC   presents a recom-mended page for review. As before, the subject is then askedto rate the usefulness of the suggested page with respect tohis/her current information needs. 4.2 Overall Results of LILAC A total of 104 subjects participated in the 5-week LILACstudy, visiting 93,443 Web pages. Over this period of time,they marked 2977 pages as IC-pages and asked for recom-mendations by clicking the “Suggest” button, 2531 times.In addition to the baseline model (FHW), we also trainedthree kinds of models, IC-word, IC-Relevant, and IC-Query,each based on browsing features. Moreover, we used thecumulative browsing logs, up to week   n −  1 , to produce theset of new models that the users downloaded for week   n . Followed Hyperlink Word (FHW)  Here, a word  w  wasconsidered important if it appeared as the anchor text of any hyperlinks that was followed in the page sequence.Notice there is no training involved in this model.This model is a baseline for this study. It was motivatedby its similarity to the “Inferring User Need by Infor-mation Scent” (IUNIS) model [6]. IC-word  We compute the “browsing behavior” features of each word  w  encountered in the current session, andalso note whether  w  appears on the associated IC-page(i.e., if it is an IC-word). We then train a decision treeto predict which words are IC-words given only theirbrowsing properties. IC-Relevant  We again compute the “browsing behavior”features of each word  w  encountered in the current ses-sion. We also explicitly ask the user to select which of these words is “relevant” to his/her search task. We thentrain a decision tree to predict which words are “rele-vant”, based only on their browsing properties. IC-Query  The IC-word model considers each word in theIC-page to be an IC-words. However, many of thesewords are are general (e.g., “the”, “page”, etc.) andtherefore not particularly relevant to the page content.In particular, only a few of these words would help lo-cate this page, in a search engine. We label these wordsas IC-Query words. To obtain these IC-Query words,we run a “Query Identifier (QI)” on each IC-page  p ,Figure 2: Overall Results of the LILAC StudyFigure 3:  WebICLite  — An Effective Complete-Web Rec-ommender Systemwhich tries to produce the subset of words that wouldlocate  p , if given to a search engine (here Google). Wethen trained decision tree to predict words most likelyto locate the IC-page by querying a search engine basedon browsing features.Attheconclusionofthestudy, wecollectedtheevaluationre-sults of our browsing feature based models (i.e., IC-models)and the baseline model (FHW). Figure 2 shows the overallresults; the bars here show the relative percentage of each of the evaluation responses for each model.As suggested by this figure, and confirmed by statistical tests(shown in “”), eachof the different IC-models perform better than the baselinemodel. This result validates our basic assumption that weare able to provide useful recommendations by integratingthe user’s browsing behaviors into the prediction. 5  WebICLite  — An Effective Complete-WebRecommender System The  WebIC   system that we used in the LILAC study hasevolved into the  WebICLite  recommendation system, whoseinterface appears in Figure 3. Like  WebIC  ,  WebICLite  is a4  client-side, Internet Explorer-based multi-tab web browser,that observes the user’s browsing behavior, extracts thebrowsing properties of the words encountered, and then usesthose browsing properties to predict the user’s current infor-mation need, which it then uses to suggest (hopefully) usefulpages from anywhere on the Web, without any explicit inputfrom the user.  WebICLite  predicts IC-pages using trainedmodels of user browsing patterns from previously annotatedweb logs. It computes browsing properties for essentiallyall of the words that appear in any of the observed pages,and generates an appropriate query to a search engine (e.g.,Google) to generate an appropriate IC-page. The query is se-lected from the predicted information need using the trainedmodels. WebICLite  differs from other IE-based browsers and Webrecommender systems as follows. Modeling (Training)  WebICLite  chooses one of severalmodels to suggest useful Web pages, where each of these models has each been trained to learn the relation-ship between user’s current page-action sequence andhis/her information needs using browsing features. Ourcurrent system then builds a general “browsing behav-ior model” from the entire population of users. (An-other option is building a personalized model for thisspecific user, based only on his/her own annotated weblogs. Here, we can obtain a recommendation that is spe-cific to the user’s specific browsing patterns, and thusmight be more accurate. We can also train the modelsfor user community based on the data that from all themembers from the same community; e.g., by country orby domain.) Recommendation  In order to generate a recommendation,the system must: First, extract the browsing features of the words in the current session, and then apply the ex-tracted patterns to generate a synthesized query, whichis subsequently sent to a search engine (e.g., Google). 5.1 Adaptive Recommender System WebICLite  can be adapted to the current browsing sessionand user. Data from the LILAC study suggests that peopletend to surf the Web by following some general browsingsession patterns. One example of a search pattern looks like: This pattern might suggest that sometimes even though theuser provides explicit input, s/he still cannot find the relevantinformation. In this case, our models might find the exactquery that will return the relevant page, from his/her brows-ing behavior.Figure 4: The Evaluation Interface of   WebICLite We found that some models work better for certain brows-ing patterns, as determined by general characteristics of thecurrent session.  WebICLite  therefore chooses the model thatworks best according to the current browsing session, thenuses this model to provide recommendations. 5.2 Evaluation Interface In the current implementation, the user can click “Suggest”to ask   WebICLite  to propose a Web page, anytime s/he needsassistance.In order to collect the relevance feedback,  WebICLite  willthen ask the user to evaluate the suggested page, as shownin Figure 4. The user will be asked to “Tell us what youfeel about the suggested page”, to indicate whether the in-formation provided on the page suggested by  WebICLite  wasrelevant for his/her search task, just as the user did in LILAC. WebICLite  also provides the options that allow the user tokeep browsing from the suggested page. He/she can selectan appropriate action:  Discard the suggested page ,  Open it on the current tab , or  Open it in a new tab . 5.3 Learning from Evaluation In order to train our models, the study participants must ac-tively label IC-pages while browsing the Web; of course, thisis inconvenient for the user, and unrealistic in a productionversion of the product. To solve this data collection prob-lem, we propose to passively train a model based on previ-ous evaluation results. Recall that every time a user requestsa recommendation, we generate a search query using one of the models to return a page to the user, which s/he is thenasked to evaluate. If we assume that the search engine (e.g.,Google) remains relatively consistent (i.e., in terms of thequality of pages returned) over time, we can infer the eval-uation of the search query from the actual evaluation of therecommended page. Thus we can label each query as oneof the evaluation outcomes. We can then attempt to learn a“Fully”-page classifier by considering (as positive examples)only the queries that are evaluated as “Fully”, and the rest asnegative.5
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks