The use of robots txt for sensitive files tutorial

The use of robots txt for sensitive files tutorial

September 3, 2017 jkdxfsgi 0

address of the background!The following

Disallow: /plus/car.php

if the site as a room in the hotel, robots.txt is the master in the room door hanging a "do not disturb" or "welcome clean" sign. This document told visiting the search engine which can enter the room and visit, what room for storage of valuables, or may involve tenants and visitors privacy without search engine open. But robots.txt is not a command, not like firewall, the gatekeeper cannot stop the thieves and other malicious intruders.

directory, you know. This program can also see ICO.

For example:

User-agent: *

Disallow: /plus/ad_js.php

There is no time to

Disallow: /plus/s>

we read, read and read the tutorial, easy to understand, in fact, is part of Shanghai dragon in Shanghai Longfeng, do people know that this file is used, in order not to let the love Shanghai spiders crawl some important page for you, or you want to love the Shanghai spiders crawl what page you, you rely on this file to control, so it gives us the convenience greatly, according to the experience of my

to combat it, let us look at the effect of

robots.txt is a protocol rather than a command. Robots.txt is the first search engine in the file when the site visit to view. Robots.txt file tells spider program on the server file is what can be viewed. When a search spider to visit a site, it will first check whether robots.txt exists, the site root directory if it exists, the robot will search range according to the contents of the file to determine access; if the file does not exist, all search spiders will be able to access the website all pages are not password protected the. Love Shanghai official advice, only if your site contains not to be included in the search engine content, only need to use the robots.txt file. If you want all the content included in the search engine website, do not create robots.txt files.

Disallow: /plus/carbuyaction.php

Disallow: /plus/advancedsearch.php

sometimes sweep all directories, can not find the background when carefully check the file, you will find this awful lot of files inside the directory, the lovely background address illustrious show in the above, you will get to visit the

Leave a Reply

Your email address will not be published. Required fields are marked *