Many people include the Robots META tag :
meta name=”robots” content=”index, follow”
In their website’s header section to encourage search engine robots to index the page (add it to their database) and “follow” all links in the page to index them as well.
While this does no harm, it does add an extra line of HTML to each page’s header section and for the interests of efficiency can be removed and replaced with a txt file called robots.txt containing these 2 lines
This file is stored in the websites root directory eg: Bhatt.id.au robots.txt and the instructions in it tell user-agents (web browsers, search engine robots etc) that all of them (* = all) are allowed to index content and follow links, and none are disallowed.