Supplementary Material to
"Simultaneous Learning and Covering
with Adversarial Noise"
Andrew Guillory, Jeff Bilmes
Introduction
This page presents supplementary material to our ICML 2011 paper
A. Guillory, J. Bilmes. Simultaneous Learning and Covering with Adversarial Noise The Twenty-Eighth International Conference on Machine Learning (ICML 2011)
The full paper is available at the author's homepage. On this page we provide links to the three different versions of the movie recommendation website we asked participants to evaluate as well as the survey participants filled out. We also give additional survey results for before and after the UI change we made during the study.
Movie Recommendation Websites
The links above are to the three different versions of the movie recommendation website we asked users to evaluate. Each version uses the same format: users are asked yes or no questions such as "Do you want to watch something from the Horror genre?" and given recommendations based on their responses. Each of the versions was generated using the worst-case greedy algorithm for noisy interactive set cover. The versions differ in the objective functions they use to choose questions and responses. The full text of the paper describes the details of each version.
Website 1 is the simplest of the three and uses objective functions which define a noise free query learning problem with membership and equivalence queries. Website 2 introduces noise into the problem so that a movie or TV show is only eliminated from consideration after two of the user's responses disagree with it. Website 3 uses more complicated objectives which take into account relationships between genres and other domain knowledge. Website 3's choice of objectives also encourages the site to recommend movies early and interleave questions and recommendations.
User Study Survey
SurveyThe link above is to the survey given to participants in our user study. The survey briefly describes the motivation behind our research and asks users to evaluate the three different sites. The sites are presented to the user in a random, user specific order using JavaScript.
Additional User Study Results
Midway through our user study we made a small UI change in an attempt to make the "More" link on recommendation pages more prominent. We made this change after observing that some users did not realize Website 3 made early recommendations which were followed by more questions and recommendations (they thought these early recommendations were final recommendations). We specifically changed the button to read "Answer More Questions for Better Recommendations" when the next page contained a question and "See More Recommendations" when the next page contained recommendations. We also added a reference to this button at the top of the recommendation page. Below are the user study results before and after this change. These tables report average results where 5=Strongly Agree and 1=Strongly Disagree. Standard deviations are shown in parentheses.
| Statement | Website 1 | Website 2 | Website 3 |
| 1 | 3.89 (1.03) | 3.50 (0.71) | 3.14 (0.90) |
| 2 | 4.29 (0.74) | 4.07 (1.09) | 4.21 (0.83) |
| 3 | 4.00 (1.12) | 3.89 (1.19) | 3.32 (1.07) |
| 4 | 3.32 (1.19) | 2.50 (0.96) | 2.82 (1.08) |
After (31 Participants)
| Statement | Website 1 | Website 2 | Website 3 |
| 1 | 3.84 (0.78) | 3.53 (1.20) | 2.97 (0.98) |
| 2 | 4.00 (0.86) | 3.94 (0.73) | 3.90 (0.79) |
| 3 | 3.90 (0.87) | 3.29 (1.07) | 3.10 (1.03) |
| 4 | 3.55 (1.12) | 3.13 (1.12) | 2.97 (1.14) |
The change did not seem to improve perception of Website 3, and the results are mostly the same. There are some differences in the average results (i.e. Website 2, Statements 3 and 4), but these differences don't seem to be related to the UI change and are most likely due to chance or demographic differences between the two participant groups. Participants in the first group were recruited from the general population and more frequently reported that they regularly used Netflix's Watch Instantly service. Participants in the second group were all recruited from one undergraduate class and less frequently reported that they regularly used Netflix's Watch Instantly.