Nitrospirae is a bot that I have written to crawl over 150+ RSS feeds and collect them in 1 single site.

Those 150+ RSS feeds are mostly IT News, PHP information, and other news which I personally favor, but it can be expanded of course, to grab many more information on the internet.

To put it simple, the purpose of this site is to gather the best of the best in 1 site and make them searchable. It can also act as a sort of IT "newspaper" which you can read everyday to get a grasp on what's going on around. You may find several articles on a same news which is written by several different sites.

http://nitrospirae.net

Please do give some suggestions or critics.

__________________
There are no noobs and pros in this world, only people who know how to use Google and those who don't.

Neat. I've been thinking about doing something along these lines for a while, but never translated those thoughts into actions.

One thing you need to do is make sure you don't get any duplicate articles on the website. When I visited I saw the same article posted on slashdot science and slashdot games.

__________________
Dingo Web Systems > http://www.dingocode.com
My Website > http://www.evanbot.com

Yeah, I have been thinking of some way to get rid of them, but some duplicates are updates made just after the script grabbed the feed, currently still finding some way to remove them perhaps...

__________________
There are no noobs and pros in this world, only people who know how to use Google and those who don't.

You could compare the URI's or flag any articles (server side, then manually check the articles if they are not too numerous) that have a high enough correlation with the similar_text() function.

__________________
My Site

Cool, I didn't know there was such function, I will look into it. Thanks for the tip!

__________________
There are no noobs and pros in this world, only people who know how to use Google and those who don't.

Just a note, the comparison algorithm is very slow O(N**3); n := str.length. This means that if a string is 5 characters long it will take 125[units], a bubble sort has a complexity of O(N**2), and a quick sort has O(n * log(n)) [best case].

What i'm saying is the comparison will take a _long_ time, so making users wait for the entire article to be scanned and then compared would be silly. I would suggest a [1,2,6,12,24] hour cron job (dependent on the server load and processing power) to check and flag/delete any similar articles.

__________________
My Site

Yes, that's what I have in my mind now.

__________________
There are no noobs and pros in this world, only people who know how to use Google and those who don't.