In search engine optimization, the typical goal is to get as many pages in your website indexed and crawled by search engines like Google.<\/p>\n
The common misconception is that doing so could result in better SEO rankings. However, that may not always be the case. Oftentimes, it is necessary to deliberately prevent search engines from indexing certain pages from your website to boost SEO. One study found that organic search traffic increased by 22%<\/a> after removing duplicate web pages, while Moz reported a 13.7%<\/a> increase in organic search traffic after removing low-value pages.<\/p>\n So how can I keep Google from indexing my website?<\/p>\n As mentioned, not all pages in your website need to be indexed by search engines. Typically, these include, but are not limited to, the following:<\/p>\n Prior to de-indexing, it\u2019s important to conduct a thorough content audit of your website so you have a systematic approach in determining which pages to include and exclude.<\/p>\n There are four ways to tell Google not to index a page in search engines: a \u201cnoindex\u201d metatag, an X-Robots-Tag, a robots.txt file, and through Google Webmaster Tools.<\/p>\n The most effective and easiest tool to disable Google indexing certain web pages is the \u201cnoindex\u201d metatag. Basically, it\u2019s a directive that tells search engine crawlers to not <\/i>index a web page, and therefore subsequently be not shown in search engine results.<\/p>\n How to add a \u201cnoindex\u201d metatag: <\/i><\/p>\n All you need to do is to insert the following tag in the <head><\/i> section of a page\u2019s HTML markup:<\/p>\n <meta name=”robots” content=”noindex”><\/b><\/p>\n Depending on your content management system (CMS), inserting this metatag should be fairly easy. For CMSs such as WordPress that don\u2019t allow users to access the source code, use a plugin like Yoast SEO<\/a>. A thing to note here is that you need to do this to every page that you wish to prevent webpage indexing.<\/p>\n Additionally, if you want search engines to both de-index your web page and not <\/i>follow the links on that page (such in the case of thank-you pages where you do not want search engines to index the link to your offer), use the \u201cnoindex\u201d with the \u201cnofollow\u201d metatag:<\/p>\n <meta name=”robots” content=”noindex,nofollow”><\/b><\/p>\n Alternatively, you can use an X-Robots-Tag, which you add to the HTTP response header of a given URL. It has basically the same effect as a \u201cnoindex\u201d tag, but with the additional options to specify conditions for different search engines. For more information, please see Google\u2019s guide here<\/a>.<\/p>\n How to add an X-Robots-Tag:<\/i><\/p>\n Depending on the web browser you use, it can be pretty tricky to find and edit the HTTP response header. For Google Chrome, you can use developer tools like ModHeader or Modify Header Value. Here are examples of X-Robots-Tag for specific functions:<\/p>\n X-Robots-Tag: noindex<\/b><\/p>\n X-Robots-Tag: googlebot: nofollow<\/b><\/p>\n X-Robots-Tag: otherbot: noindex, nofollow<\/b><\/p>\n A robots.txt file is mainly used to manage search engine crawler traffic from overloading your website with requests. It must be noted, however, that this type of file is not meant to hide web pages from Google; rather, it is used to prevent images, videos, and other media files from appearing in search results.<\/p>\n How to use robots.txt file to hide media files from Google: <\/i><\/p>\n Using robots.txt is fairly technical. Basically, you need to use a text editor to create a standard ASCII or UTF-8 text file, and then add that file to the root folder of your website. To learn more on how to create a robots.txt file, check out Google\u2019s guide here<\/a>. Google has also created separate guides for hiding certain media files from appearing in search results:<\/p>\n You can also choose to temporarily block pages from Google search results using Google Webmaster\u2019s Remove URLs Tool. Please note that this is only applicable to Google; other search engines have specific tools of their own. It\u2019s also important to consider that this removal is temporary. For permanent removal of web pages from search engine results, view Google\u2019s instructions here<\/a>.<\/p>\n How to use Google Remove URL tools to temporarily exclude pages:<\/i><\/p>\n The procedure is quite easy. Open the <\/span>Remove URLs Tool<\/span> and select a property in the Search Console that you own. Select <\/span>Temporarily Hide <\/b>and enter the page URL. Afterwards, choose \u00a0<\/span>Clear URL from cache and temporarily remove from Search<\/b>. This hides the page from Google search results for 90 days, and also clears the cached copy of the page and snippets from the Google index. For more information, check out <\/span>Google\u2019s guide here<\/span><\/a>.<\/span><\/p>\n It may take time for Google to receive your request for de-indexing. It often takes a few weeks for the change to kick in. If you notice that your page is still appearing in Google’s search results, it’s most probably because Google hasn’t crawled your site since your request. You can request for Google to recrawl your page using the Fetch as Google tool.<\/p>\n If you want to know more, or if you need help with any of your SEO needs, Ilfusion has the expertise and experience to lend you a hand. Give us a call at 888-420-5115<\/a>, or send us an email to creative@ilfusion.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"In search engine optimization, the typical goal is to get as many pages in your website indexed and crawled by search engines like Google. The common misconception is that doing so could result in better SEO rankings. However, that may not always be the case. Oftentimes, it is necessary to deliberately prevent search engines from… view more<\/a>","protected":false},"author":1,"featured_media":7594,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,1280],"tags":[1502,633,74,1501,1500,134,19],"class_list":["post-7593","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articles","category-seo","tag-crawl","tag-google-search","tag-google","tag-index","tag-metatags","tag-search-engines","tag-seo"],"yoast_head":"\nWeb Pages That Don\u2019t Need to Be Indexed<\/span><\/h5>\n
\n
How to Stop Google Indexing a Page<\/span><\/h5>\n
1. Using a \u201cnoindex\u201d metatag<\/h6>\n
2. Using an X-Robots-Tag HTTP header<\/h6>\n
\n
\n
3. Using a robots.txt file<\/h6>\n
\n
4. Using Google Webmaster Tools<\/h6>\n
Wrapping It Up: How Can I Keep Google from Indexing My Website?<\/span><\/h6>\n