Yahoo, Google and Microsoft Team Up on Sitemaps

By
Yahoo, Google, and Microsoft have all announced that they’ve agreed to set a standard for sitemaps.

DiggSpeak Translation: "Amazing! Top Ten Reasons to use Sitemaps"

Sitemaps are those XML files that list all the pages on your Web site. Search engines like to have all the listings in one place so that a site can be indexed without anything being missed.

The protocol has now been released under Creative Commons, so any search engine can pick up on it if they like.

Most webmasters / developers and web site owners use sitemaps, and there is plenty of sample code to generate these dynamically.

We use sitemaps on our Eggheadcafe.com site, and I believe they result in much better indexing. Plus, you can specify how often the bots should crawl, and what the priority is of each item. For more complex sites, you can have a SiteMapIndex file in your website root, which has entries that point to any number of other individual sitemap files. So for example, you might have a messageboard or Forum section and create a separate sitemap for that out of your database daily. Then, you might have another sitemap for your "regular" content such as articles, that gets updated weekly. Your index file would point to both of these, and the bots will happily crawl them.

Sitemap files can also be GZipped, which cuts down on bandwidth and the bots can load them faster. With .NET, just use the System.IO.Compression namespace.

The nice thing about sitemaps is this - Bots only know how to do one thing: follow links. They can't follow Javascript; they can't follow images, and they can't follow dropdownlists of links either. It has to be an anchor "A" tag, with an "href" attribute to make the spiders happy. Yep, they are just "reallyreallydumb".

If you have content in your site that's not linked to from another page, or which only comes up because of a database search, it's not likely to get indexed at all. But if you put the url into a sitemap file, the bots will find it, crawl, and index it. Think about it - you may have content that is dynamically generated out of a database, and doesn't sit on the filesystem at all. With the correct sitemap element, that content, which ordinarily would be invisible to the spiders, will be successfully crawled and indexed by the major search engines. That means more hits, and more revenue if you serve advertising.

Sitemaps are your friend. Now, with Yahoo and Microsoft on the bandwagon, they'll be more important than ever.

Yahoo is expected to begin using your sitemap(s) on Thursday, with Microsoft picking them up early next year.
Peter has been active on the C# and other MS newsgroups, has had samples at gotdotnet.com (now codegallery.com) that have been downloaded over 56,000 times, and has written numerous articles for eggheadcafe.com, for which he is a co-founder, and MSDNAA sites. He has also been a technical reviewer on books from Prentice Hall, Manning and other publishers. Peter was a Microsoft MVP for 10 years. Peter was born and raised in New York City, and after a stint as a jazz musician playing contrabass with the Robert Hunt Trio, migrated to sunny Florida, where he was hired as a rookie Financial Consultant at Merrill Lynch in Orlando, specializing in Corporate Services. It was at Merrill that Peter realized computers and technology were his forte. He spent his last few years at Merrill developing real – time computerized trading and and stock charting program using advanced technical analysis indicators, some of which are his own inventions. In 1997, he moved on to architect and serve as lead developer on a number of innovative web-based and middleware solutions for the healthcare and financial services industry.

Comments

  1. News is Good3:13 PM

    I bet sidemaps aren't at all important in any way for anybody who isn't hopelessly unalive.

    ReplyDelete
  2. Hong Xiaowan3:31 PM

    Sitebases, the next protocol after Sitemaps
    It can save the time and press for the search engine, also for the websites.
    It can bring new search engine that named Search Engine 2.0.
    Using Sitebases protocol, will save 95% bandwidth above. It is another sample for long tail theory.
    In this protocol, I suggested that all search engine can share their Big Sitebases each other, so the webmaster just need to submit their Sitebases to one Search Engine.
    And I suggested that all search engines can open their search API for free and unlimited using.

    Please visit: http://www.sitebases.org

    ReplyDelete
  3. peterbromberg6:23 PM

    Hong,
    I looked at your post and I didn't see any protocol, or API, only a short description. You don't explain how "SiteBases" can save 95% bandwidth.

    Moreover, what incentive do big search engines, which all compete with each other for advertiser dollars, have to share their result databases with each other?

    Finally, most big search engines have already opened their APIs for use by developers. Not "Unlimited" use, but enough to do some good things.

    Maybe you could follow up with some detailed responses to my 3 points?

    ReplyDelete
  4. Hong Xiaowan8:58 AM

    Hi Peter.
    1.For "about", please click http://www.sitebases.org/about-sitebases/; for protocol http://www.sitebases.org/protocol/. It is almost the same with sitemaps.org. The search engines maybe treat http://www.sitebases.org as mirror as http://www.sitemaps.org but their different. Just like checks, the same looking. But one is 100USD, another is 10000USD.
    2.My target is to save 95% hardware cost not only bandwidth cost. The webmasters submit their sitebases including whole content of their websites. The search engine just need to select some pages to check if same as Sitebase. But no need to download whole site. Please see http://www.sitebases.org/protocol/. That will be help you to understand my ideas.
    3.Sharing the big sitebases will change all. I just said "should" not "must". It is new idea that can save much more money. No matter how many search engine want to protect their Sitebases as private, the newer will open and comes to the next.
    4.The advertiser will use more platform like adsense, adwords. These platform can be new field.
    5.Yes, now the limited API still enough. But sometime back busy not as original search engines.

    All will change.

    ReplyDelete
  5. Hong Xiaowan10:25 AM

    I like to talk with you for this topic.

    ReplyDelete
  6. Anonymous2:28 PM

    My email is hongxiaowan(at)gmail.com
    I am glad to discuss some topic with you. Thanks a lot.

    ReplyDelete

Post a Comment

Popular posts from this blog

Some observations on Script Callbacks, "AJAX", "ATLAS" "AHAB" and where it's all going.

By
I've taken a more than cursory interest in the whole Remote Scripting (.NET species) vs. AJAX and now ATLAS discussion, mostly because I started using Remote Scripting since Microsoft first released it, and because I continued to refine it after seeing Brent Ashley's excellent work with JSRS, which was one of the first "real" cross-browser solutions back in 2000 (that's the turn of the Century for you history buffs). Now Bertrand Leroy, a Microsoft guy whose work I like , has authored some very interesting ASP.NET 2.0 script callback stuff that he points to on his blog, and he is obviously (at least, to me he is) involved quite heavily in the development of this new ATLAS infrastructure that they'll preview at PDC. Leroy's most recent post brings to the surface what I agree are the major differences between the "AJAX a - la .NET" approach as popularized by Michael Swartz with his AJAX.NET library, and Leroy's RefreshPanel, the ASP.NET 2.0 b...
Read more

IE7 - Vista: "Internet Explorer has stopped Working"

By
Image
This one was a bitch. All of a sudden for no reason at all, out of the blue, I get this dialog "internet Explorer has stopped working". I didn't install any new software, crap- I didn't do anything! So here I am using FIREFART to go on the internet and find out what to do! Damn! Good thing I've got the sucker on the same machine (I'm not "anti-Firefox", i just don't use it that much except to check my page renderings). The Fix The fix (at least for me): 1) Go into Control Panel, and choose "Internet Options". 2) Under the "Advanced" tab, press the "RESET" button at the lower right: Don't ask me why this happens, or why the fix works. That's a BUG, D00D - I don't care how you slice it!
Read more

FIREFOX / IE Word-Wrap, Word-Break, TABLES FIX

By
One of the most annoying things a developer (who would normally delight in writing code, not futzing with markup) can have is getting rendering issues between different browsers. The two biggest players are of course Internet Exploder and Firefart (as I lovingly like to refer to each). In most cases that's going to take care of 98 percent of your total site traffic. IE has had a proprietary "word-break" style attribute for a long time, and this made it into the CSS 3.0 spec. But that doesn't necessarily help you with Firefox right now. Firefox literally - (and I consider this completely idiotic, since it's been in their bug database for FIVE YEARS) does not have a reliable CSS style element to force table cell content to break in the middle of a word in order to stop the content from expanding your table / div off the page or over other content on the page. Here's a partial fix, which will work for most tables and browsers. The style declaration (I call it m...
Read more

System.Web.Caching.Cache, HttpRuntime.Cache, and IIS Recycles

By
Recently I chimed in on a newsgroup thread to an OP (Original Poster) who was questioning why he was losing Cache items. This individual would load a dataset from the Cache object into a local object to use to perform processing. He had been using httpcontext.current.cache and noticed that sometimes he was not able to get the dataset out of the cache. He tried switching to HttpRuntime.Cache and it seemed to help. The OP claimed that the application is under heavy hits, and once or twice every month he would see this error. He wanted to know why does the HttpContext.Current.Cache call work sometimes, while some other times does not. A couple of posters replied, one of whom is a respected MVP who usually makes very astute, high-quality posts. He posted," The cache is only valid during the life of the request. how is you[sic] code using the cache?" Somebody else (also pretty astute, usually) responded, "Cache is valid during lifetime of application not request.....". W...
Read more

FIX: Requested Registry Access is not allowed (Visual Studio 2008)

By
Symptom: in Visual Studio 2008, you attempt to add a new WebContentForm and associate it with a MasterPage. You receive an error dialog "Requested Reqistry Access is not allowed". Don't ask me what caused this; I'm guessing that the registry keys involved had their access denied due to removal of a previous version of Visual Studio, leaving the new guy (Visual Studio 2008) in the lurch. Here's the fix: Download "subinacl" here . This is a command-line tool that enables administrators to obtain security information about files, registry keys, and services, and transfer this information from user to user, from local or global group to group, and from domain to domain. After installing subinacl.exe (it will be in the C:\Program Files\Windows Resource Kits\Tools folder.), drop to a command prompt and run the following batch file, which you have saved in the same folder with the filename "FIX.bat": subinacl /subkeyreg HKEY_CLASSES_ROOT\VisualStudi...
Read more