Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sn_master
on March 27, 2021
|
parent
|
context
|
favorite
| on:
Only Google is really allowed to crawl the web
Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: