New! SiteMap xml AutoDiscovery with Robots.txt

Apparently Google, Yahoo and MSN all getting together about agreeing to use the Sitemap xml format to index your web site is having some good fallout.

They've all agreed to look at your robots.txt file (which all crawlers look for) and you can now point to your sitemap url from there, and they will happily chew upon it!

To do this, simply add the following line to your robots.txt file:

Sitemap: http://www.example.com/sitemap.xml

You need to provide the complete URL for your Sitemap on this line. They will pick it up --wherever you put it in your robots.txt file. This directive is not specific to user-agent. If you have multiple Sitemaps, you can point to your Sitemap index file on this line.

Details about the Sitemaps protocol including this addition are available on the protocol website here: http://www.sitemaps.org.

The Google sitemap invention and protocol is a great SEO tool for your web site or blog. We have used this on our eggheadcafe.com site since the inception of sitemaps. On my IttyUrl.net site, i actually have a page that reads and caches my own autogenerated sitemap xml and turns it into a "page of links" - just so non-sitemap bot crawlers have something to consume.

Now that MSN, Yahoo and Google are all digesting this stuff, you've got yourself a triple with the same amount of effort as just getting on to first base.

Comments

  1. NATO245:05 PM

    Do you think that crawlers would be put off if the file suffix is .ashx as apposed to .xml?

    Thanks,
    Nathan

    ReplyDelete
  2. peterbromberg5:43 PM

    I really am not sure. Here's the spec:

    http://www.sitemaps.org/protocol.php

    ReplyDelete
  3. NATO246:36 PM

    Thanks, no mention in the specification that it needs to conform to a specific file name.

    Did you create a job to generate the sitemap.xml file on ittyurl.net or is it generated dynamically?

    Thanks for your help and all your excellent article; you've help me many many times over the years.

    ReplyDelete
  4. peterbromberg7:21 PM

    It depends on the site. At IttyUrl.Net, all the content that would normally go into a sitemap is in my database. I have a method called from Application_Start in global.asax that checks the last time a sitemap was generated. This then gets the data I need through a complex SQL join statement that is passed to my custom SiteMapGenerator class which generates and deposits a new sitemap.xml file from it.
    Hope that helps.

    ReplyDelete

Post a Comment

Popular posts from this blog

Some observations on Script Callbacks, "AJAX", "ATLAS" "AHAB" and where it's all going.

I've taken a more than cursory interest in the whole Remote Scripting (.NET species) vs. AJAX and now ATLAS discussion, mostly because I started using Remote Scripting since Microsoft first released it, and because I continued to refine it after seeing Brent Ashley's excellent work with JSRS, which was one of the first "real" cross-browser solutions back in 2000 (that's the turn of the Century for you history buffs). Now Bertrand Leroy, a Microsoft guy whose work I like , has authored some very interesting ASP.NET 2.0 script callback stuff that he points to on his blog, and he is obviously (at least, to me he is) involved quite heavily in the development of this new ATLAS infrastructure that they'll preview at PDC. Leroy's most recent post brings to the surface what I agree are the major differences between the "AJAX a - la .NET" approach as popularized by Michael Swartz with his AJAX.NET library, and Leroy's RefreshPanel, the ASP.NET 2.0 b
Read more

IE7 - Vista: "Internet Explorer has stopped Working"

Image
This one was a bitch. All of a sudden for no reason at all, out of the blue, I get this dialog "internet Explorer has stopped working". I didn't install any new software, crap- I didn't do anything! So here I am using FIREFART to go on the internet and find out what to do! Damn! Good thing I've got the sucker on the same machine (I'm not "anti-Firefox", i just don't use it that much except to check my page renderings). The Fix The fix (at least for me): 1) Go into Control Panel, and choose "Internet Options". 2) Under the "Advanced" tab, press the "RESET" button at the lower right: Don't ask me why this happens, or why the fix works. That's a BUG, D00D - I don't care how you slice it!
Read more

FIREFOX / IE Word-Wrap, Word-Break, TABLES FIX

One of the most annoying things a developer (who would normally delight in writing code, not futzing with markup) can have is getting rendering issues between different browsers. The two biggest players are of course Internet Exploder and Firefart (as I lovingly like to refer to each). In most cases that's going to take care of 98 percent of your total site traffic. IE has had a proprietary "word-break" style attribute for a long time, and this made it into the CSS 3.0 spec. But that doesn't necessarily help you with Firefox right now. Firefox literally - (and I consider this completely idiotic, since it's been in their bug database for FIVE YEARS) does not have a reliable CSS style element to force table cell content to break in the middle of a word in order to stop the content from expanding your table / div off the page or over other content on the page. Here's a partial fix, which will work for most tables and browsers. The style declaration (I call it m
Read more