(2)
Almost every webmaster, especially at the initial stage of creating a project, faced the need to remove website pages from the search engine index. Even despite the fact that the procedure, at first glance, is quite simple, many still have difficulties.
Sometimes the owner of a web resource has to hide a document from search engines for the following reasons:
- the site is under construction and got into the SERP quite by accident;
- the content on the page is no longer relevant;
- the document duplicates another page that is already on the site;
- the index includes service pages where the personal data of clients are located.
To avoid such cases, today we will talk about 4 effective ways to remove a page from the search engine index.
How to close a page from search engines using the webmaster's panel?
This method of blocking access to the pages of your site for search robots is considered one of the easiest. Moreover, this tool is ideal for those cases when certain URLs need to be removed urgently.
Yandex
To do this, you need the Yandex.Webmaster service. We have told you how to add a site to this site to speed up indexing. Follow the link https://webmaster.yandex.ru/tools/del-url/ and add the address of a specific page in the corresponding field, then click “Delete”. 
With a high degree of probability, Yandex will ask you to speed up the process of deleting a page from the system database. To do this, you need to first close it from bots through a file or a robots meta-tag, or make the server return a 404 error. We will talk about how to do this a little later.
It will take several hours or even days before the bots delete the document from the database. This is due to the fact that the system will need to track its status and make sure that it does not change anymore.
Log in to the site Google Webmaster Tools... Add your site to the index first if you haven't already. Then find the "Google Index" tab there, and under it "Remove URLs". In the window that appears, select the option to create a delete request, and enter the address of the document to be deleted in the field. Then submit your request.
Server 404 error
Surely every user, looking for the necessary information on the Internet, ended up on the page where he gave error 404 - "Page not found"... This means that the document you were looking for was removed from the resource.
The webmaster can do this in the site control panel, for example,. For search engines, this means that you configure the response server of this page so that the code appears at a specific address 404 notfound... And when the robot visits the given URL again, the server will inform him that the page no longer exists. This makes it clear to search engines that the document has been removed from the site, and they will remove it from the search results so that visitors do not go to it and read the 404 error.
The characteristic features of this method include:
- Easy setup in just a few clicks.
- Complete disappearance of the document from the web resource. Because of this, it is not recommended to use this method when a service page (confidential information of clients, etc.) needs to be removed from the index.
- It is also worth resorting to another option for hiding the page, for example, a 301 redirect if incoming links lead to it.
Important! The page falls out of the search index not due to its removal from the resource, but due to further reindexing. Therefore, to remove it, you will have to wait about 2 weeks until the bot visits the resource again.
For webmasters, this method is one of the most convenient, but the visitor may not like the 404 error, and there is a risk that the user, seeing it, will stop visiting the site. But there is a way out of this situation as well.
On a note. Very often, site builders are engaged in an interesting design of the page on which the 404 not found error crashes. They post useful information there and offer to visit other pages of the resource, which will certainly attract the attention of the visitor. This will make this page more attractive to the user, which will certainly have a positive effect on his ranking and recognition.
Modifying the robots.txt file
Another common method used by webmasters. It allows you to hide individual documents and entire sections. In the robots file, you can not only deny, but also allow search bots to index a site or certain pages. Such commands are provided by the Disallow directive.
To hide a page from search engines, you need to access the root folder of the site. The robots.txt document basically contains 2 lines:
- User-agent. The name of the robot of a certain search engine, which you prohibit crawling the page, is indicated here, or the code is written User-agent: *, applicable to all bots at once.
- Disallow. The URL of the page to be deleted is written here.
Together they create a command to search engines for the specified URL. If required, you can hide several documents on one resource in one file at once, and they will not affect each other in any way.
For example, this is how we tell all search robots not to index the index and password pages on our website.
User-agent: * Disallow: / index Disallow: / password
When you write this command, you will need to wait for the next indexing. You should know that all changes to the robots file are advisory in nature for search engines, so do not be surprised if the object still remains in the index, but is marked to be hidden through robots.txt. But if after a certain period of time the status of the document does not change, it will still be deleted from the search engine database.
Important! If there are external links on the remote pages, then they will remain available for clicking on them. This is due to the fact that the commands in the robots file do not actually remove the object from the search results, but only hide it from the results.
Robots Meta Tag
If you do not go into details, then this method is similar to the above, only all commands are written in the html-code of the site inside the head tags:
All changes will also be made after indexing. The advantage of using the meta robots tag is that you can remove multiple URLs from your search using it without having to change the robots. This method is ideal for novice site builders who gradually create new pages.
It is also a great option when you need to close a page from the Yandex index, but at the same time keep it available on Google. Or when you want to remove an item from the search, but leave the information available on the site itself.
An example of closing a URL for Yandex only:
Important! Writing meta tags differs from changing robots.txt in that you can use them to remove an object from search, even if external links lead to it. This is facilitated by the noindex meta tag. However, if your site is not built on WordPress, then the procedure will be more complicated. The fact is that WP has a special Yoast SEO plugin that allows you to delete pages with meta tags.
Conclusion
If you suddenly need to remove a specific page of your site from the search index, you can easily use all the methods indicated in this article. Choose the one that is more convenient for you, but at the same time take into account the characteristic features of each of them. In general, if you want to close the document as soon as possible, use several options at once.
Sometimes you need to remove a site page from the search results, for example, if it contains confidential information, is or has been removed.
Tell Yandex to delete the page
This can be done in several ways:
If the page is removed from the site
- Disallow indexing of the page using the Disallow directive in your robots.txt file.
Configure the server so that when the robot accesses the page address, it sends an HTTP status with a 404 Not Found, 403 Forbidden, or 410 Gone code.
If the page should not appear in the search
- Disallow indexing of the page using the Disallow directive in your robots.txt file.
- Close pages from indexing using the noindex meta tag.
If you have customized the server response to 4XX code, or used the noindex meta tag for multiple pages, the robot learns about each page change incrementally. In this case, disable indexing of pages using the Disallow directive.
When a Yandex robot visits the site and learns about the ban on indexing, the page will disappear from the search results within a week. After that, the address of the deleted page will be displayed in the list of excluded pages in the Indexing → Pages section in Yandex.Webmaster search.
For some time, the robot will still visit the page to make sure that its status has not changed. If the page remains inaccessible, it will disappear from the robot base.
The page may reappear in search results if you remove the robots.txt file from indexing or the server response changes to 200 OK.
If a page is removed from the site due to copyright infringement, the order of its removal from search results does not change and does not take precedence.
Speed up page removal from search
If, after you have disabled indexing of pages, the robot has not yet visited your site, you can tell Yandex to remove the page using the Remove pages from search tool in Yandex.Webmaster.
Remove individual site pages from Yandex search
add and confirm the site in Yandex.Webmaster.
Make sure your robots.txt file is set to Disallow for the pages you want to remove. If the robot finds other directives for the pages specified in Yandex.Webmaster in robots.txt, then, despite them, it will remove the pages from the search.
You can delete a directory, all site pages or pages with parameters in the URL. To do this, add and confirm the site in Yandex.Webmaster.
In Yandex.Webmaster, go to the Tools → Remove pages from search page.
Make sure your robots.txt file is set to Disallow for the pages you want to remove. If the robot finds other directives for the pages specified in Yandex.Webmaster in robots.txt, then, despite them, it will remove the pages from the search.
- Set the switch to By prefix.
- Specify a prefix:
You can send up to 20 directives per site per day.
- Click the button Delete.
Statuses after URL submission
How to return a page to search results
Remove the directives prohibiting indexing: in the robots.txt file or the noindex meta tag. Pages return to search results when the robot crawls the site and learns about the changes. This can take up to three weeks.
We have released a new book, Content Marketing on Social Media: How to Get Into the Heads of Subscribers and Fall in Love with Your Brand.
To achieve your goals in working with the site, you need to take into account a lot of different subtleties. During inattentional optimization, it happens that the index contains information that is not for the eyes of surfers of the Internet. It is logical that you need to get rid of this kind of content. In this article, we will analyze the common cases when such a need arises and, of course, we will learn how to remove a page from Yandex search.
Reasons for removing web pages from search
We will not discuss the problems of the giants, such as the story of the well-known mobile operator, whose name I will not advertise, I will just say that it starts with “mega” and ends with “background”. When the latter got indexed and got into the search for a page with SMS messages from users. Also, we will not focus on the problems of online stores, where you can periodically find the personal data of users and the details of their orders in the public domain. And so on.
We will analyze the painful problems of ordinary entrepreneurs who want to solve their problems:
- Doubles. Duplicate content interferes with the promotion of sites, this is no secret to anyone. The reasons for the occurrence can be different. More details can be found in our .
- The information has lost its relevance. For example, there was some one-time action, but it passed. The page needs to be deleted.
- "Secrets". Data that's not for everyone. We recall the situation with the telecom operator. Such pages must be hidden from onlookers.
- Changing URL addresses. Whether it's moving a site or setting up a CNC, in any case, you will have to get rid of old pages and provide the search engine with new ones.
- When moving to a new domain name, you decided to remove the site from the Yandex search engine altogether so that the uniqueness of the content would not be affected.
In fact, there can be many more reasons to delete saved pages from search engines. The list is only needed to illustrate typical problems.
How to remove a page from Yandex search engine
There are two ways of development of events. Long (but lazy) and accelerated (you have to move a little).
- Long way
If time does not rush and there are more important things to do, then you can simply delete the page through the content management system (it is the admin panel or CMS). In this case, going to a non-existent URL, the search robot will find that the server responds with a 404 code - the page does not exist, which means it needs to be thrown out of the search, which will eventually happen without your intervention.
- Accelerated way
A kind of continuation of the first. After removal from the CMS, you need to use the Yandex service remove url ... This will lead to a prompt reaction from the PS and in the near future unnecessary information will disappear from the index.
But what if you need to remove a page from the search engine, leaving it on the site itself?
How to delete a page in Yandex search engine without deleting it from the site
First, block the page from indexing using the robots. txt, I'm sure you've already encountered it. Inside the file, you need to write the following:
User-agent: Yandex
Disallow: / i-hate-my-page
This will block content from indexing, the URL of which is www.domain.ru / i-hate-my-page
Secondly, since robots. txt does not contain a strict set of rules for the robot, but rather is of a recommendatory nature, then the usual Disallow may not be enough. Therefore, for greater confidence, write the following in the html code of the page:
Note that the head tags are not here for show. The robot meta tag must be placed between the tags
.The culmination will be the addition of the address to the already familiar Yandex service Remove URL, which we already know how to use.
How to remove a site from Yandex search engine
In order to completely remove a site from the search base, you need to go through the same steps as for deleting a page, but with some nuances.
- Close the entire site from indexing by adding to the robots. txt lines
User-agent: Yandex
Disallow: / - Remove pages using the native Remove URL service.
- Get rid of all links leading to your site.
- Wait for reindexing.
If you no longer need the site at all, delete all the files that are on the hosting and forget about it. As a last resort, contact the PS support service.
Outcome
I would like to say that the procedure for removing saved pages from the search is not very difficult and sometimes it is simply necessary. Hopefully, site owners who do not want to go deeply into the details will be able to find in this article the answer to the question of how to remove unnecessary web pages from Yandex search.
Hope it was helpful!
Hello! Today's post is about painful for most of the novice site builders. I had to answer the same question very often in the comments - how to remove pages from search that were indexed earlier, but due to the current circumstances were deleted and no longer exist, but are still in the index of search engines. Or there are pages in the search that are prohibited from indexing.
In the comments, you will not expand much, therefore, after the next question, I decided to pay special attention to this topic. First, let's figure out how such pages could appear in the search. I will give examples based on my own experience, so if I forget something, then please add.
Why are closed and deleted pages in the search
There may be several reasons, and I will try to highlight some of them in the form of a small list with explanations. Before starting, I will give an explanation of what I mean by "extra" (closed) pages: service or other pages that are prohibited from indexing by rules or a meta-tag.
Non-existent pages are searched for the following reasons:
- The most commonplace is that the page has been deleted and no longer exists.
- Manual editing of the address of a web page, as a result of which a document that is already in the search becomes unavailable for viewing. Particular attention should be paid to this point for beginners who, due to their little knowledge, are dismissive of the functioning of the resource.
- Continuing the thought about the structure, let me remind you that by default, after installing WordPress on a hosting, it does not meet the requirements of internal optimization and consists of alphanumeric identifiers. It falls on the CNC, while a lot of non-working addresses appear, which will remain in the index of search engines for a long time to come. Therefore, apply the basic rule: if you decide to change the structure, use a 301 redirect from old addresses to new ones. The ideal option is to complete all the site settings BEFORE opening it, a local server may come in handy for this.
- The server is not configured correctly. A non-existent page should return an error code of 404 or with a 3xx code.
Extra pages appear in the index under the following conditions:
- The pages seem to be closed, but in fact they are open to search robots and are searched without restrictions (or the robots.txt is not spelled correctly). To check PS access rights to pages, use the appropriate tools for.
- They were indexed before they were closed in accessible ways.
- These pages are linked from other sites or internal pages within the same domain.
So, we figured out the reasons. It is worth noting that after eliminating the cause, nonexistent or extra pages may remain in the search base for a long time - it all depends on the frequency or frequency of the robot's visit to the site.
How to remove a page from the Yandex search engine
For remove URL from Yandex just follow the link and in the text field of the form insert the address of the page that you want to remove from the search results.
The main condition for a successful delete request:
- the page should be closed from indexing by robots rules or a noindex meta-tag on this page - if the page exists, but should not participate in the search;
- when trying to access a page, the server should return a 404 error - if the page is deleted and no longer exists.
The next time the robot crawls the site, the delete requests will be executed and the pages will disappear from the search results.
How to remove a page from Google search engine
To remove pages from, we do the same. Open Webmaster Tools and find Remove URLs in the Optimization drop-down list and follow the link.
Before us is a special form with which we create a new delete request:
Click continue and following the further instructions, select the reason for the deletion. In my opinion, the word "reason" is not quite suitable for this, but that is not the point ...
Of the options presented, we have available:
- deleting a page page from Google search results and from the search engine cache;
- deleting only the page from the cache;
- deleting a directory with all addresses included in it.
A very convenient function for deleting an entire directory when you have to delete several pages, for example, from one category. You can monitor the status of the deletion request on the same page of the tools with the option to cancel. For a successful deleting pages from google the same conditions are required as for. The request is usually completed in no time and the page disappears from the search results immediately.
