Where to put robots.txt for crawler?

Support for webhosts that use CentOS
User avatar
WhatsHisName
Posts: 1546
Joined: 2005/12/19 20:21:43
Location: /earth/usa/nj

Re: Where to put robots.txt for crawler?

Post by WhatsHisName » 2008/11/24 23:30:11

Paraphrasing Ned...

From [url=http://en.wikipedia.org/wiki/Robots.txt]wikipedia.org - Robots Exclusion Standard[/url]:

[quote][b]The protocol ... relies on the cooperation of the web robot, so that marking an area of a site out of bounds with robots.txt [i]does not guarantee privacy[/i].[/b]

[b]Some web site administrators have tried to use the robots file to make private parts of a website invisible to the rest of the world, [i]but the file is necessarily publicly available and its content is easily checked by anyone with a web browser.[/i][/b][/quote]

MajorNewbie
Posts: 127
Joined: 2008/04/17 00:04:43

Re: Where to put robots.txt for crawler?

Post by MajorNewbie » 2008/12/01 02:20:04

Has anyone uploaded product attributes to Google Base and is willing to share the know-how?

The basic Google Base product taxonomy is not adequate for my purpose and I would like to expand it as per http://base.google.com/support/

Also I am uncertain in formatting the attributes, for example, if I am selling a kettle, should I explicitly insert a comma, like this:
-
Electric Kettle (Black, 1.5 Quart)
http://www.mywebsite.com/product/electric-kettle-15l-44.cfm?sid=google
electric kettle water boiler teapot
http://www.mywebsite.com/media/images/JS202BLKx270%2Ejpg
20601
1.00

44
MasterCard
Visa
[color=990000] [b]electric kettle, electric water boiler, teapot [/b][/color]


The general idea is to direct people to my site through Google when they key in "electric kettle" or just "kettle" eg.
[color=990000] [b]electric kettle, kettle, electric water boiler, water boiler, teapot [/b][/color]

Post Reply

Return to “CentOS 5 - Webhosting Support”