When I worked in e-commerce as a SRE, bots were doing two things:
- trying to disrupt business processes (eg: false referral listings, gift card scams, etc)
- trying to disrupt systems
I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.
There's also a fair amount of scraping for things like...
- Reselling aggregated data
- Competitive pricing and inventory data
- "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve
- Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it.
- And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
> I'm sure there are folks who use bots and scrapers for home automation
I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.
I don't think anyone mentioned crawling (in my mind, "crawling" refers to a long-running process over several pages or sites and "scraping" could be a single fetch of a single URI)
I'm sure there are others, just from the top of my head:
* Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating.
* Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability
* Public transport: next bus/trains from closest station, delays and interruptions
* IMDB/tvdb/etc for monitored shows and movies. Common with sonarr.
Some people have smart mirrors or any other kind of 'news display', so it may be useful for them to scrape the data they may think relevant (this may be weather data, stocks, or even the new Nintendo Switch availability at their nearby retailer).
For whatever reason in years of running a SaaS this only happened maybe 3 times and never with my online shop. I guess using stripe and a few basic security settings keeps them away mostly
- trying to disrupt business processes (eg: false referral listings, gift card scams, etc)
- trying to disrupt systems
I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.