How to tell search engines not to look at this specific link?

Member

by stephon , in category: SEO , a year ago

How to tell search engines not to look at this specific link?

Facebook Twitter LinkedIn Telegram Whatsapp Pocket

2 answers

by kyleigh.wolff , a year ago

@stephon 

To tell search engines not to crawl a specific link, you can use the "robots.txt" file. This is a text file placed on your website's server that specifies which pages or sections of your site should not be crawled by search engine bots. You can add the following line to your "robots.txt" file to block a specific URL from being crawled:

1
2
User-agent: *
Disallow: /path/to/specific/page


Note that this method is not 100% foolproof and determined crawlers may still attempt to access the restricted pages, but most search engines will respect the rules specified in the "robots.txt" file.

by genevieve_boehm , a year ago

@stephon 

To tell search engines not to look at a specific link, you can use a "noindex" meta tag. This tag can be placed in the HTML header of a webpage and tells search engines not to index the page or follow its links. Here is an example of how you can use the noindex tag:

1
2
3
4
5
6
7
8
<html>
<head>
  <meta name="robots" content="noindex">
</head>
<body>
  ...
</body>
</html>


Keep in mind that not all search engines honor the noindex tag, and some may still index the page despite it being specified. Additionally, it may take some time for search engines to reflect the change, so it may still appear in search results for a period of time after the noindex tag has been added.