Daniel Zhou 周晓丹 Mens sana in corpore sano

This paper is submitted to ACM Recommender System Conference'09, co-authored with my advisor Prof. Paul Resnick.

ABSTRACT

Conversation double pivots recommend target items related to a source item, based on co-mentions of source and target items in online forums. We deployed several variants on the drupal.org site that supports the Drupal open source community, and assessed them through clickthrough rates and subjective assessments of the relevance of the recommendations. Overall, clickthrough rates increased by a factor of 15 from the beginning to the end of the tuning process. A similarity metric based on correlation of mentions rather than mere co-occurrence reduced the problem of over-recommending the most popular modules, but additional corrections for recency and uniqueness of mentions were not helpful. Detection of more module mentions in conversations dramatically improved the quality of recommendations, even though the detection algorithm then had more false positives. Recommendations based on conversation co-mention were more effective than those based on co-installation, because co-installation data only led to recommendations of complementary modules and not substitutes.

Attached is the full-paper.

Discussion of algorithm improvement: http://drupal.org/node/479812

(due to un-controllable spamming, comments were disabled temporarily. sorry...)

There's a big demand from the Drupal community to add fivestar-like ratings to the contrib modules. This would be a pretty cool feature, but it has other concerns too.

In a paper presented in WWW'04 conference titled "Shilling recommender systems for fun and profit" (PDF), the researchers talked about different ways and examples of gaming recommender systems. Such things could happen because the hackers what to 1) promote certain items, 2) "nuke" certain items, and 3) disrupt the entire systems. The researchers also simulated 2 kinds of automatic attacks, NormalBot and AverageBot, and showed that recommender systems are indeed vulnerable to such manipulation attacks.

Apart from academic research literature, it's easy to imagine possible manipulation on Drupal module ratings, either for fun or for profit. New modules are especially vulnerable because a few initial false negative ratings could very likely prevent further evaluations. In fact, if you search "rating manipulation" in Google, you'll find eBay, imdb, Amazon, etc are all victims. And I heard the module rating system in Joomla suffered the same thing too.

To cope with this issue, we need more sophisticated algorithms. My advisor, Paul Resnick, is one of the leading researchers in this area. He and Prof. Rahul Sami published a paper proposing a manipulation-resistance algorithm (PDF. However, a simpler and more intuitive alternative might be to maintain two rating scores -- one from all users, one from the experts. And the experts could be defined as users who registered for more than 1 year and submitted issues, etc.

To sum up, in this blog I'm trying to argue:

1. Module rating system for drupal.org would be helpful,
2. But the concern of gaming the system is real and legitmate
3. Measurements should be taken if the rating system is deployed.

My summer funding would probably come from an NSF research grant on manipulation-resistant algorithms. I'll try to make some proposals on how to prevent gaming the module rating system, if d.o. infrastructure team decides to implement it, after I read more papers in the area.

From the experience of developing the "pivots" Drupal module recommendation system, I developed the general purpose Recommender API module. It was released today.

In fact, if you think about it, all recommender systems work pretty much the same way. Take Amazon for example, it knows what users bought what items, and then it can calculate how similar users' tastes are, and finally make recommendations based on that. Take "pivots" (based on project_usage) for another example. It knows what sites are using what modules, and then calculates related modules based on the fact that related modules are used in the same sites.

In Drupal, we have a lot of such relations as the users-items relation in Amazon, or the sites-modules relation in pivots. For example, we have the nodes-votes relation in VotingAPI, or the users-products purchasing relation in Ubercart, or the nodes-terms relation in Taxonomy. You name it. Those relations can be used to calculate similarity among items, and then generate recommendations.

The question is how to calculate similarity among items, and how to generate recommendations. That's where the Recommender API module comes into the picture. It provides a set of general purpose APIs that take into such users-items relation as input, and then calculate similarities and predictions. The module itself doesn't have any end user interaction, but it provides powerful APIs for other module developers to write cool recommender-based modules.

My hope for this module is that it could give rise to a wave of new modules development involving recommendations, e.g. Ubercart purchase recommendation, personalized ads, etc. I also wrote a simple example module User-to-user Recommendation to show how easy it is to use the API. I'm hoping Content Recommendation Engine and Similar By Terms modules could also take advantage of this module too, and support other recommender algorithms that might generate better results in different cases.

The Recommender API would be devoted to provide more algorithms (like PageRank, SVD, PCA, etc.), more features (like manipulation-resistance, etc), and enhanced performance.

You comments/feedback would be much appreciated :)

From the last Google Analytics (GA) study on the usefulness of "pivots_block" on 4 recommendation algorithms, we learned that the classical "relevancy" algorithm generated the better results. Therefore, we used the relevancy algorithm on D.O. from Dec/4/2008 to Jan/9/2009. And the average click-thru rate was 0.474%.

Then, we were interested in whether 0.474% was indeed a good indicator to claim "pivots_block" was useful to users. So we temporary switched "pivots_block" to the "New forum posts" on the module pages, and try to compare if the "New forum posts" could invite more clicks. It turned out that the average click-thru rate for "New forum posts" between Jan/9/2009 and Jan/18/2009 was only 0.144%. That means, more or less, on the module pages "pivots_block" is almost 3 times more useful than "New forum posts".

Now we are much more confident that "pivots_block" and its intended goal of providing module recommendations is indeed useful to the community. So we are going to switch back to "pivots_block", and continue to make improvements to it.

One thing we learned from this study was that users seem to like more "serendipity" in the "pivots_block". One problem for the current algorithm is that the recommended modules for a particular module remains quite stable over time. And we are going to try out an approach of adding a little bit variability to the recommendations.

The new algorithm is coming soon!