使用PHP在RSS源中查找“类似”文章

Question

There is something I am trying to accomplish although I'm not really sure where to start.

I currently have a MySql database with a list of articles. The DB contains the article title, content, and some other info like dates, etc.

There is an RSS feed that we monitor for new articles, it's a Google Alert feed that just contains the latest news on certain subjects. I want to be able to automatically monitor this feed and record any feed items that are similar to stories currently in our DB.

I know how to set a script to run automatically, and I know how to parse the RSS feed with SimplePie.

What I need to figure out is how to take the description of the rss feed items, run a check on our DB to see if the feed item is similar to something we have in our DB, and return a numerical score of some sort, sort of like a "similarity rating" or something.

After that I can have the info I need recorded to the DB if the "similarity rating" is above a set limit, which I know how to do.

So my only issue is how to compare each feed item to our current articles, and return a score based on how similar it is.

Answer 1

The Levenshtein function (available for both PHP and MySQL) is a good way to handle this. It basically calculates a value based on the number of permutations (replacements, moves, etc) required to convert one string to another. That score would be your "similarity rating".

EDIT: the Levenshtein function is not available natively in MySQL but there are SQL implementations of it that you can use such as: http://kristiannissen.wordpress.com/2010/07/08/mysql-levenshtein/