To block access to your site for all robots and search engines ("Disallow all"), all you need is a robots.txt file containing these two lines:
User-agent: *
Disallow: /
In short: place a robots.txt at the root of the site containing User-agent: * and Disallow: /. Only robots that respect this protocol follow it: to really close access, password-protect the site with Folder protection.
Prerequisites
- Being able to place a
robots.txt file at the root of your site (by FTP or with the file manager).
What does this file do?
The first line means that the instruction below applies to all robots. The second line blocks everything accessible within the domain: all directories, folders, pages and files.
This rule is only followed by robots that respect the robots.txt protocol (Google, Bing, etc.). Malicious robots or those not following robots.txt rules may still access the site's content.
Does Disallow prevent pages from being indexed?
The Disallow instruction prevents crawling of the site, but not the indexing of its pages. Even with a total Disallow, Google may therefore index an address it knows from an external link, without reading its content.
To remove pages from search results, do the opposite: let robots crawl those pages (no Disallow) and add the meta robots noindex tag to them. A robot that does not crawl a page never reads its noindex tag: the two methods therefore cannot be combined on the same page.
How do I really close access to the site?
To be certain of blocking the entire site, protect it with a password. The Apanel console does this without any file to write: in Hosting management, Website section, open Folder protection, protect the root of the site and add a user in the Restrictions by users tab. The server then answers with an HTTP 401 code ("Unauthorized", authentication is required to access the resource) to any visitor without credentials, robots included.
Troubleshooting
- The file seems to be ignored: check that it is named exactly
robots.txt, in lower case, and that it is shown at https://your-domain/robots.txt.
- Pages remain in Google's results despite the Disallow: this is the case described above. Remove the Disallow on those pages and use the
noindex tag, or close access with a password.
Going further