"Our goal here is to be able to distinguish between the good type and bad type of scraping and give webmasters full transparency. Obviously this is a hard problem. If you have any feedback on any of this we'd love to hear it."
Yes as said before plus:
- Obey robots.txt to the full extend
- Name your access, i.e. label your bot
- Don't use shady tactics such as IP rotation
- Provide web site owners the option to fully block access
of your bots (yes, communicate your full IP ranges)
Again - this is from a content owner who paid for his content.
Yes as said before plus:
- Obey robots.txt to the full extend
- Name your access, i.e. label your bot
- Don't use shady tactics such as IP rotation
- Provide web site owners the option to fully block access of your bots (yes, communicate your full IP ranges)
Again - this is from a content owner who paid for his content.