function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
rumblerrumbler 

robots.txt and force.com Sites

There are a number of "special" files that must be located in the web root of a public website including:

/robots.txt - Controls web search engine behaviors
/favicon.ico - Icon for website

There doesn't appear to be any way to specify the values for these files. Salesforce has provided a default robots.txt which prevents search engine indexing, but production orgs will need the ability to override this file. Also the location of favicon.ico can be set using meta tags, but the default location is still used by some older browsers, so there should be a way to assign an image to this file. I would hope these two issues could be addressed before production roll out.

George
Best Answer chosen by Admin (Salesforce Developers) 
BulentBulent
You can upload your custom robot.txt via metadata API. Here are the steps:

1. Install the force.com ide

2. Create a Visualforce Page that represents the robots.txt you would like

<apex:page contentType="text/plain" showHeader="false">
User-agent: msnbot
Disallow: /
</apex:page>

3. Add the following to your package.html

<types>
<members>*</members>
<name>CustomSite</name>
</types>

4. "Refresh from Server" before you can see the sites.

5. Open up your site under the "sites" folder and add in a "robotsTxtPage" entry and save.

<?xml version="1.0" encoding="UTF-8"?>
<CustomSite xmlns="http://soap.sforce.com/2006/04/metadata">
<active>true</active>
<authorizationRequiredPage>Unauthorized</authorizationRequiredPage>
<bandwidthExceededPage>BandwidthExceeded</bandwidthExceededPage>
<changePasswordPage>ChangePassword</changePasswordPage>
<fileNotFoundPage>FileNotFound</fileNotFoundPage>
<inMaintenancePage>InMaintenance</inMaintenancePage>
<indexPage>IdeasHome</indexPage>
<masterLabel>First</masterLabel>
<portal>Customer Portal</portal>
<requireInsecurePortalAccess>false</requireInsecurePortalAccess>
<robotsTxtPage>RobotsTxt</robotsTxtPage>
<siteAdmin>mylogin@myco.com</siteAdmin>
<subdomain>mydomain</subdomain>
</CustomSite>

6. Now http://mydomain.force.com/robots.txt will contains the description of your page


You can also use your own custom favorite icon. Here are the steps:

1. create 16x16 icon
2. upload as a static resource and make sure cache control attribute is set as public
3. Disable the standard header in your visualforce page and reference the favorite icon static resource in your page as following:

<apex:page showHeader="false" >
...
...
<link REL="SHORTCUT ICON" HREF="{!URLFOR($Resource.favicon)}">
.....
</apex:page>

All Answers

BulentBulent
You can upload your custom robot.txt via metadata API. Here are the steps:

1. Install the force.com ide

2. Create a Visualforce Page that represents the robots.txt you would like

<apex:page contentType="text/plain" showHeader="false">
User-agent: msnbot
Disallow: /
</apex:page>

3. Add the following to your package.html

<types>
<members>*</members>
<name>CustomSite</name>
</types>

4. "Refresh from Server" before you can see the sites.

5. Open up your site under the "sites" folder and add in a "robotsTxtPage" entry and save.

<?xml version="1.0" encoding="UTF-8"?>
<CustomSite xmlns="http://soap.sforce.com/2006/04/metadata">
<active>true</active>
<authorizationRequiredPage>Unauthorized</authorizationRequiredPage>
<bandwidthExceededPage>BandwidthExceeded</bandwidthExceededPage>
<changePasswordPage>ChangePassword</changePasswordPage>
<fileNotFoundPage>FileNotFound</fileNotFoundPage>
<inMaintenancePage>InMaintenance</inMaintenancePage>
<indexPage>IdeasHome</indexPage>
<masterLabel>First</masterLabel>
<portal>Customer Portal</portal>
<requireInsecurePortalAccess>false</requireInsecurePortalAccess>
<robotsTxtPage>RobotsTxt</robotsTxtPage>
<siteAdmin>mylogin@myco.com</siteAdmin>
<subdomain>mydomain</subdomain>
</CustomSite>

6. Now http://mydomain.force.com/robots.txt will contains the description of your page


You can also use your own custom favorite icon. Here are the steps:

1. create 16x16 icon
2. upload as a static resource and make sure cache control attribute is set as public
3. Disable the standard header in your visualforce page and reference the favorite icon static resource in your page as following:

<apex:page showHeader="false" >
...
...
<link REL="SHORTCUT ICON" HREF="{!URLFOR($Resource.favicon)}">
.....
</apex:page>
This was selected as the best answer
rumblerrumbler
Bulent,

Thanks for the information on robots.txt, that was very helpful. With regards to favicon.ico, you are correct that one can use a shortcut tag but I occasionally still see requests to websites requesting /favicon.ico even with the tag present. I'm no expert on the behavior of IE 6 and other older browsers with respect to favicon.ico, but it may be nice to be able to map /favicon.ico to a static resource or document just to cover all of the varied browser behaviors out there.

Thanks,

George
BulentBulent
Thanks for the suggestion.
We are planning to add the favicon lookup to the site setup.
cherialcherial
I have  a custom object Student__c & an html page Student_Entry.html

Student__c has two fields: Admission_Number__c & Student_Name__c
Student_Entry.html page has two text fields Admn_num & stud_name and a Submit button.

Suppose when a user fills the form and click Submit button then i want to insert a new record into the Student__c
object with the details taken from the form.

Plesae suggest me a step by step approach to achieve this. If possible suggest me resources too...

Thanks in advance,
-Nath
Ron HessRon Hess
The step by step starts with
  1. create a visualforce page (and possibly controller ) that performs that action.
  2. expose that visualforce page by adding it to your Force.com sites list of visualforce pages allowed.

If you have any trouble creating your visualforce page, please check out the Visualforce documentation, or post questions and code to the Visualforce developer board.
Disha GuptaDisha Gupta
Hi, can we edit default robots.txt file which is found at your "SalesforceOrgURL/robots.txt"? I need to remove contents from its allow section. Please let me know if there is any way to do this.