Resources for Robots txt [1 - 1 of 1]
  • Robots-txt-generator 1.01

    robots-txt generator The spiders that visit your site consumes much of your bandwidth? If you want to control the robots or spiders that visit your site, robots-txt generator is the solution. It is very common that the webmaster of each site spends a lot of money paying for bandwidth and it is unfair that a significant portion of that bandwidth is consumed spiders. Robots txt generator, you can stop or control the bots. Some robots or spiders saturate your server making countless queries per ...

    Robots txt | | 51KB | 2009-04-17 04:25:14