Update 2/7/2013: Question and answer now available for updates/edits at "Why aren't search engines indexing my content?"

The topic of why particular sites/sites' pages are not being indexed comes up from time to time and, rather than closing questions as too localized, it would benefit the community to have a troubleshooting Q/A catch-all to address common misconceptions and causes for indexing issues.

Title:

Why aren't search engines indexing my content?

Question:

My site (or specific pages on my site) is not appearing in search engine results.

Why isn't my content indexed and what can I do about it?

Answer:

There are a number of reasons your content may not appear in search engine results, however, it is important to note that a search engine's index may contain pages that it doesn't display in its results page.

How to tell if your content is actually indexed

It may actually be difficult to tell if your content is indexed.

  • Search for all the documents from your site and see how many are listed
    • Google: enter site: example.com (where example.com is your domain)
    • Bing: enter site: example.com
    • Yahoo: enter site: example.com (or use advanced search form)
  • Search for a specific document by a unique sentence of eight to twelve words and search for that sentence in quotes. For example, to find this document, you might choose to search for "number of reasons your content may not appear in search engine results"
  • Log into webmaster tools to see stats from the search engine itself about how many pages are indexed from the site
    • Google Webmaster Tools -- Information is available under "Health" -> "Index Status". If you have submitted site maps, you can also see how many documents in each site map file have been indexed.
    • Bing Webmaster Tools

In some cases, documents may not appear to be indexed via one of these methods, but documents can be found in the index using other methods. For example, webmaster tools may report that few documents are indexed even when you can search for their sentences and find the documents on the search engine. In such a case, the documents are actually indexed.

How Content Gets Indexed

Before search engines index content, they must find it using a web crawler You should check your webserver's logs to see whether search engines' crawlers (identified by their user agent - e.g. Googlebot, Bing/MSNbot) are visiting your site.

Larger search engines like Google and Bing typically crawl sites frequently, but the crawler may not know about new site. You can notify search engines to the existence of your site by registering as its webmaster (Google Webmaster Tools, Bing Webmaster Tools) or, if the search engine does not provide this facility, submitting a link to its crawlers (e.g. Yahoo).

How long has your site/content been online?

Search engines may index content very quickly after it has been found, however, these updates are occasionally delayed. Smaller search engines can also be much less responsive and take weeks to index new content.

If your content has only been online for several days and does not have any links from other sites (or its links come from sites which crawlers do not visit frequently) it is probably not indexed.

If your site hasn't been live for more than a few months, the search engines may not trust it enough to index much content from it yet.

Has the content been excluded?

This step is especially important if you are taking over a site from someone else and there is an issue with a specific page or directory: check for robots.txt and META robots exclusions and remove them if you want crawlers to index the content being excluded.

Is there a technical issue preventing your content from being indexed?

If you have an established site but specific content is not being indexed (there are no web crawler hits on the URLs where the content resides) the webmaster tools provided by Google and Bing may provide useful diagnostic information.

Google's Crawl Errors documentation provides extensive background on common problems for web crawlers which prevent content from being indexed and, if you use Google Webmaster Tools, you will receive an alert if any of these issues are detected on your site.

Correct errors and misconfigurations as quickly as possible to ensure that all of your site's content is indexed.

Please update this question/answer pair if anything is lacking - if everything looks good, I will post it up as a community wiki later this week.

share
3  
Good work danlefree. –  paulmorriss Feb 5 '13 at 11:35
    
Is it worth suggesting a site: or info: search to see if pages are indexed? A number of questions have asked "Why isn't my site indexed", when in fact their site is indexed but is ranking poorly (if at all) for a particular keyword search. Also (although this is perhaps covered by the "technical issues/crawl errors")... make sure pages return a 200 HTTP status code. Again, with a couple of questions, the problem was that their pages were erroneously returning 404's, "but the page looks OK". –  w3d Feb 5 '13 at 17:49
    
@w3d I was thinking about including the site: operator suggestion but decided in favor of recommending Google Webmaster Tools - it offers a much deeper look into the issue (including HTTP status code diagnostics) and it provides a better idea of what's indexed. site: leaves the questions of whether something is indexed but not yet in the SERP's and whether the content is accessible to Googlebot unanswered. –  danlefree Feb 5 '13 at 20:07
    
@danlefree we might as well add the site: search, because some people might not be able to add webmaster tools to their site (they don't have root access, etc.) –  Christofian Feb 6 '13 at 0:06
    
@Christofian If they cannot verify by any of the means available, (file upload, META tag, DNS record, Google Analytics snippet) wouldn't that leave in question whether their question concerns "the operation of websites which [they] control" ..? –  danlefree Feb 6 '13 at 1:53
    
@danlefree what if they're using something like wordpress.com (I'm not sure if people can add webmaster tools to a wordpress.com site, but for the sake of the argument, let's assume that they can't). They would still have control over much of the website (they could control the content, the appearance, etc.), and a lot of the answers on the site would be applicable to their website (how to format titles, how to get traffic, etc.). They just wouldn't have 100% control over their website. –  Christofian Feb 6 '13 at 23:08
    
@Christofian alright, that's a strong a case for including those instructions. Done. –  danlefree Feb 7 '13 at 5:40
    
Q&A pair posted - I'll start a quick sweep for duplicates to close out. –  danlefree Feb 7 '13 at 17:16
add comment

You must log in to answer this question.

Browse other questions tagged .