Keith Hogan – Ask.com

• Less than 35% of servers have a robots.txt file
• Most are copied from one found online
• Typical 23 character (100 is about the max)
• Format is not well understood
• May change to xml format for better control in the near future
• Ask: can find info about a crawler on site in the about page.

Eytan Seidman – MSN Live Search

• Used example www.hilton.com/robots.txt (tells them not to use it during the day… funny)

Dan Crow – Google

• Also need to focus on robots exlusion – robots.txt + robots meta tags
• Exclusion Protocol: tells search engines what not to index
• Comparted to sitemaps which tells search engines what to crawl
• Search engines have a lot of differences between them
• There is interest to standardize protocol for all search engines

Sean Suchter – Yahoo

• Yahoo slurp is the web crawling robot
• Yahoo! Slurp = user-agent identifier
• Supports all standard robots.txt commands (robotstxt.org
• Custom
• • Crawl delay
  • Sitemap – specify location
  • Wildcards – specify patterns of urls to disallow/allow
  • Custom meta extensions – NOODP and NOYDIR – do not use Yahoo directory titles
• Please only address yahoo robot with that section in .txt file
• Currently supports crawl-delay
• • Often misused
• Microformats.org/wiki/robots-exclusion
• • Demark sections of html you don’t want robots to use for matching
  • Used to demark useless template text, ad text, etc.. irrelevant traffic
• Historically – to identify what pages to not show in search results
• But… there is more beyond those (css, images, inline text, iframes), do we need a mechanism to exclude that in certain respects?

Danny Sullivan – Host

• Check out webmasterworld’s robots.txt file – has lots of notes
• Wonders if robots.txt in XML format to make it easier
• • Maybe combine the sitemap and robots to one file – check and prevent in one shot

]]> http://www.morpheusmedia.com/mlog/archive/ses-nyc-2007-robotstxt/feed/ 0 http://www.morpheusmedia.com/mlog/archive/due-to-seo-abuse-wikipedia-nofollows-links/ http://www.morpheusmedia.com/mlog/archive/due-to-seo-abuse-wikipedia-nofollows-links/#comments Wed, 31 Jan 2007 22:33:25 +0000 Toby Evers http://www.morpheusmedia.com/mlog/?p=10 As reported by Search Engine Land here, Wikipedia has put “no follow” tags on all it’s outbound links. Gray and Black hat SEO’s have been using Wiki to create “hot” inbound links to their sites that they are trying to promote. Even with all of the algorythm changes the search engines have made, inbound links from high ranking sites are still holding their stature. Since Wikipedia is a well traveled site, it just seems natural to promote a site via spamming links all over this “open source” site. So, once again, the site gets spammed and has to ruin it for others by putting a no follow tag on all links. What this means, is that when a crawler makes it’s way to a Wikipedia article, it will not grab the link and follow it to the site it’s linking to, rendering this “gray/black” hat SEO technique useless.