5/25/2005

FeedSearchGator Up and Running!


FeedSearchGator is my newest creation based on the fact that there are now literally dozen of search engines that return results as RSS.

What FeedSearchGator does, very simply, is to take your search query that you put on the QueryString, send it out to currently eleven (11) different engines on a threadpool asynchronously, aggregate the results, remove duplicate links, and return a single RSS feed containing up to several hundreds search results that you can subscribe to in your RSS Feed reader. The engines are: feedster.com, search.news.yahoo.com, beta.search.msn.com/news/, moreover.com, blogdigger.com, waypath.com, blogpulse.com, rocketnews (news), rocketnews (blogs) , newsisfree.com,and daypop.com. I am giving my custom threadpool a 20 second WaitAll timeout since occasionally, one or two of these seach engines take a long time to return results (a very long time, too long to wait for a page).

I'm still refining this, but it is fully operational and I'm open to ideas (including marketing ideas) for the concept. The feed results are cached for 12 hours by query, so if your query is something new it can take up to 20 seconds to get back results. Obviously, if your search term is one that's already been run in the last 12 hours, you'll get your results much faster.

I strip out unneccessary HTML tags to keep the "glop" and the bandwidth down, and results are sorted by most recent pubDate first. I've also put in an REL link to a simple XSL transform in the RSS Document which means you can also view this in your browser too.

Currently the engine is set up to insert up to 3 custom advertisements as actual RSS entries, although this feature is currently turned off.

So for example, if I wanted a custom feed on "XMLHTTP" i would point my RSS Reader to:

http://www.myidentityplus.com/feedsearchgator/default.aspx?q=xmlhttp

Enjoy!