No spider ever has indexed every link on the Internet. The Internet is growing so fast that it is difficult for any to keep up. There is a strong chance that every page on YOUR website will be indexed, and that every link on each of your Web pages will be followed.

Generally, when a new site is launched, a search engine spider will index the home page only. On subsequent visits, the spider will go deeper and deeper into your site, over a period of time that is determined by the number and quality of links from other websites that point to your website. Getting your site listed in the major directories will accelerate the time it takes to have your site fully spidered. Remember, links from directories are links to your website. Google specifically recommends submitting your site to Yahoo and to The Open Directory if you're having trouble getting indexed by the Google spider.