Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: