> I need to make a general remark to people who are evaluating (and/or) planning to introduce anti-bot software on their websites. Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks.
If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository.
None of these anti-bot providers give a shit about invading your privacy, tracking your every movements, or whatever other power fantasy that can be imagined. Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.
I wish they'd limit it to just stopping credential stuffing.
Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving up some JavaScript to check my browser.
The information I'm trying to get is public and can be accessed without any kind of authentication. I'm willing to bet that they flipped the anti-bot stuff on their load balancer on for the entire site instead of doing the extra work to only enable it for just electricitycompany.com/myaccount/ (where you do have to log in).
I also asked the company if they'd be willing/able to push the power rates out via the smart meters so that my interface box (Eagle-200) could pick it up, they said they have no plans to do so.
The next step is to scrape the web site for the provincial power regulator, which shows the power rates for each provider. Of course, the regulator's site has different issues (rounding, in particular), I haven't dug any further to see if I can make use of this.
All of this effort to get public information in an automated fashion.
At a minimum any scraper that doesn't execute JS needs to impersonate a screen reader user agent. Locking out disabled people has to be many levels of illegal in most countries.
As a disabled person myself I will go as far to suggest that the websites should allow unfettered bot access to the disabled i.e. anything a user without malicious intent is allowed to do on a platform should be allowed to be done by a bot for a disabled because accessibility & equity are a joke.
Social media platforms has made physical appearance as the first class citizen of the reputation economy. I'm not even talking about those platforms which outright bury content from the disabled as a policy, I'm talking about those platforms whose algorithms favor selfies, videos over text/URLs and thereby putting those with accessibility issues in severe disadvantage.
Why would you use such platforms one might say, Do something which has nothing to do with the reputation economy they might add; Well have you looked at LinkedIn lately? LinkedIn has become ubiquitous with professional job search and 30 second video intro is the very first thing on the profile, not the skills which the platform was meant to be when it was launched. One must be naive to claim that the physical appearance on that video or profile picture doesn't affect the job prospects(Several studies have stated otherwise).
It's not just the physical appearance, The action of creating videos or posting photos itself is hard as a time-constrained person[1] and so I think it's reasonable to ask the platform to allow bots to post deep-fake videos of the user doing silly things which these platform expects from an average user.
Blocking for not supporting JS isn't illegal nor a violation of the US ADA. You can add requirements for disabled people to access your services as long as it's reasonable, and the prevalence of screen readers that work with JS turned on likely qualifies requiring JS a reasonable request. It'd be like saying "you can't deny someone using a IE8 screen reader by only offering TLS 1.3".
> Unfortunately, the days of reliable non-JavaScript capable scraping are over.
Not really. In a lot of cases websites use JavaScript to call some API along with some on the fly generated token to prevent abuse.
As long as that token isn't captcha you can reverse engineer the site to do scraping without javascript and that is so much faster than browser based scraping.
I agree with this. This is what I see on a lot of sites I scrape. Reverse engineering the JS to figure out how the fuck the token was generated is a bitch though.
So then you use headless browsers to render the js and that is even hackier, but totally worth it to hit another full webpage request to get the token, so you can go back to plain requests.
I don't think token is only the thing it comes to play here. If the company wants they can use various other techniques like fingerprinting, tls fingerprinting and lot of thing.
Its just a cat and mouse game. After few year I think hardware attention etc will come to play which can mitigate bot issue somewhat.
I’m not sure what rate you are trying to get but the electric market in the US has 5 minute settlement periods. So for your region you would need to grab the price for each period and average that to get a power rate. Take that rate and add transmission fees, taxes and various other fees your provider tacks on then multiply that by usage. In Texas you can go directly to the ERCOT site and get these prices and not worry about counter measures. I’m not sure where you are but there is likely a similar whole sale site that you can access.
I had never heard of this, but it looks like a reasonable option. My go-to for this type of thing would be Python+Selenium+Firefox, but only due to familiarity with those.
Playwright is easy to get started with. The even tools that allow you to record your browser actions and covert it into code ( https://playwright.dev/ ).
Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.
In some cases it is because the consumer has opted for variable rates, essentially making a bet that net-net, variable rates will be less expensive than the fixed rates. Or that they would be able to shift usage to reduce usage during spikes. See: https://www.texastribune.org/2021/02/22/texas-pauses-electri...
Feb 2021 FT:
"Bills mount in Texas power market after freeze sends prices soaring:
Financial casualties emerge as grid operator Ercot requires billions in payments
"
In Alberta, the electricity system has been deregulated so you can buy from numerous providers. The Utilities Consumer Advocate shows 187[0] different electricity plans available in my city. My currently plan and provider changes rates monthly, but some providers allow you to sign up for 3-year or 5-year fixed-rate plans.
Thanks to deregulation, the electricity rate isn't the only thing you pay for though. There is also a Transmission Charge, Distribution Charge, and Local Access Fee. These are all per-kWh charges and change very rarely.
In October, my electricity rate is $0.10730/kWh, but my total cost is actually $0.16346/kWh plus the per-day charge ($0.202/day). Tomorrow the November rate will be published.
This also depends on the country. Where I live (Europe) the rate now changes by the minute, or thereabouts. That was made possible after everybody had to change to wireless meters. Sometimes you'll get a warning in advance - a newspaper may write "If you live here or here, don't do your cooking at this particular hour". Some providers still have fixed rate options, some apparently don't. What I dislike the most is that they're trying to force us to run our washing during the night, something the insurance company and the fire department warn intensely against. And I don't want to be at sleep if a fire starts (which happens here and there, through the year). But that's what the pricing scheme tries to enforce.
I’m curious to see the stats they are relying on and the communications materials the fire department and insurance company are using on this topic.
It seems to me from a life-safety angle that their energy would likely be far better spent on recommending smoke alarms, CO meters, and periodic cleaning of dryer vents than on recommendations against sleeping with washing/drying machines running.
They do all of that as well, of course. The problem is that when it does happen (and statistically, it will, somewhere, at some point) there's a chance you don't hear the alarm (very common - just this morning there was a newspaper story about someone who were saved by the neighbours, they didn't wake up right away even with all the alarms blaring. What the alarms did though was to alert the fire department, as per their setup).
In short - if there's a fire it's much better that you're awake and up already.
Octopus Energy in the UK has a tariff that charges half-hourly rates. They also offer an API to interrogate current pricing and usage and encourage their customers - including domestic users - to take advantage of it:
https://developer.octopus.energy/docs/api/
Bots aren't just trying credential stuffing. They are:
- committing clickfraud to game ad and referral revenue systems
- posting fake or spam reviews and comments
- generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments
- validating stolen credit card details
- screwing with your metrics collection if you can't identify them as bots
All of that is enough reason for sites to use bot detection and blocking technology. The fact that the same tech also has some utility against accidental or malicious traffic-based DoS is also a bonus.
To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they test it with, so why not pick something that they'll personally benefit from? There are so many online games with purchasable currency these days, it's hard to choose.
Not exactly. Some people are in the business of gathering and selling valid credit cards.
They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.
Oh, for sure. It's definitely not either/or. I've worked in fraud prevention software in the past and our clients would definitely see both.
I've been away from that world for a while but remember that more serious operations will separate the cashing out part (either money or goods) from their acquiring / validating operation because the former carries more risk.
There's also an interesting episode of the darknet diaries podcast (https://darknetdiaries.com/episode/85/) about card cloning which I found interesting.
Doing tiny transactions to validate cards is a thing. I've seen this happen, it's a known problem, and that at other times it's larger transactions does not make it go away.
> How do you validate the card is live? Run a real transaction through it and mark it based on the result.
Not necessarily. Some online services, particularly shops that do home delivery, may give their customers the possibility of adding a card to their wallet and perform a verification as part of the process. As a result, it becomes possible to validate a stolen card number without performing an actual transaction.
My bank called as they had red flagged a suspicious charge on on my credit card. The sum was $0. The bank rep told me that this often flies under the radar and doesn't show up in some records and it indeed did not show up in my transaction history that i could see online in my banking details. But yeah the point was exactly the same. The fraudsters testing whether the charge goes through and the card is alive/valid.
Did you verify that it really was the bank calling and not a scammer? I get calls and texts from “credit card fraud departments”, “banks”, “service warranty departments”, “Amazon billing”, “Microsoft security”, “Social Security Administration”, “IRS”, and others frequently; 95% are scammers.
The most amusing to me are the ones from “Microsoft” to alert me that they have detected malware on my computer.
If you're trying to fight financial crime it's important to not simplify behavior seen in the wild. Like the other commented noted, there is a very clear difference in behaviour between those validating cards and those using cards in order to steal.
All very valid reasons (except to a degree the "screwing with metrics" one). There a lot of websites which do not really face any of the aforementioned issues, simply because they do not sell you anything, are not an ad network or run referral programs, do not even have user-generated content, etc. And even the sites that do, it's usually a rather small part of the "surface" that needs such protections e.g. the actual API call to make a checkout or post a comment.
And yet if you dare browse the web with TOR or a VPN or sometimes just happen to be on a small ISP[0][1] then you're being punished immediately. You solve your cloudflare-supplied captcha because you may be a bot (you're not, and the dangerous bots will not be defeated by this anyway, but some humans will be), and then you get an error from the website itself because it runs a secondary bot detection thing. And you weren't even anywhere near anything "dangerous" like a checkout page.
[0] My parents use a regional small ISP (but locally very popular) that serves around 50K customers. My parents also use a regional bank (a Volksbank, and those are members of a national association that provides all kinds of services). Suddenly that bank would not even let them see the bank's front page. After some back and forth on the phone support line it turned out the bank had recently deployed some "advanced" bot detection, one that had a whitelist of residential-ISP-associated AS/IP ranges, and of course whoever compiled and maintained that list had forgot to include that small local ISP. For that regional bank it meant they had just shut out a very significant number of their customers (and potential customers just trying to look up what the bank offers), as there was very likely a huge overlap of people using that regional ISP and that regional bank (both are regional, after all). It also was something they couldn't fix themselves, as the "online banking" stuff was not in-house but was run by the national association (which probably used some bot-detection as a service provider). It took the bank (or rather, the national association) a few weeks to fix. Mind you, the last few years that bank has been heavily marketing a cheaper "online only" account, only online banking, online support, and access to the self-serve ATMs and banking terminals, but no face-to-face or even ear-to-ear human interaction. Try contacting "online support" about "the website outright refuses me" when the "online support" is only available on that website. Kudos is you're smart enough to switch to their mobile app, as your phone uses a different ISP, unless you forget to turn off wifi. That's the advice my parents got from the phone support (they sill have an account type that not online-only).
[1] When I recently visited my parents, from the wifi [same ISP as in 0] I couldn't open the website of a bakery too look up if and when they would be open on a Sunday. Some error message about "this website is not available in your network" (English text, for a German bakery... suspicious :P). I could open it via my mobile, tho. I could open it from my regular ISP when back home (in another city) again. Mind you, that website is purely informational and has no "interactive" features let alone let's you buy anything. It's just static text and some pictures.
It doesn't help that VPN providers in the consumer space pay small ISPs to front their traffic so that netflix et al doesn't drop their traffic coming from a datacenter and they still get to keep up the false pretenses and get valued at billions of dollars.
Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services which invite people to sell their residential bandwidth in return for money, this can potentially have a disproportionate impact at smaller sample sizes.
Networks can declare themselves to be ISPs. You could check if your ISP shows up as an ISP in peeringDB.
Yeah, but that's not what happens with this small German ISP my parents use, which was started because some people and local businesses really got disgruntled at the pricing policy and lack of true broadband by the Deutsche Telekom. This thing is essentially run like a non-profit, the little they make in profit meant to be invested into the company again not to make some investors happy, and is majority owned by the city-owned municipal utilities company, with the rest of the ownership I believe in the hands of some local businesses (some of which quite large) who needed broadband but couldn't get it or only at astronomical prices; they are their own customers and thus not very much inclined to fuck themselves.
They are reportedly very proactive when it comes to CPE security, as well, up to giving customers a proactive phone call when they see somebody is using equipment with known vulnerabilities (customers are allowed to operate their own equipment as long as it is deemed compatible, most will use remotely managed equipment, tho, I believe; my dad used to use his own DSL router and once got such a call if I remember correctly. He switched over to their fiber now and managed equipment).
Their AS is indeed identified as an "ISP" in peeringDB.
While I would be extremely surprised if the company was doing shady things, you surely got a point that a small ISP like that could suffer more in reputation from some few customers being up to shady things, including sub-lending the line. I am pretty sure that is against the ToS, but enforcement is a problem of course. Especially detecting such traffic without violating German privacy laws is probably a difficult task, but it's not impossible.
Yeah, I used to work for one of the major anti-bot vendors. Customers weren't clueless. Nobody buys these solutions because they're so much fun, it's a cost center and they monitor their ROI quite closely. Credit card charge backs, impact to infrastructure, extra incurred cost due to underlying api's (like in the Airline industry in particular) etc are all reasons why bot mitigation is a better option than nothing for a lot of companies, even if it's not 100% effective.
You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me (grumble why does it even need to ask for my desktop's accelerometer data via js...)
Stuff like this is a pain beyond pain. I really hope that the clients you mention know that they piss off a proportion of their users with every move they take.
> I really hope that the clients you mention know that they piss off a proportion of their users with every move they take.
With all due respect, if the tech can make a large impact on the problems mentioned above, I'm sure it's an easy decision for the big companies to take decimating bot activity over the tiny minority of users who proactively decide to disable JavaScript.
You could always go into your local bank branch instead of accessing it over the Internet. Your desktop's accelerometer helps add to your computers 'run by a human' score. Normally I'd take more issue with whatever possible privacy issue there, but my bank is where I keep my money so I'm really okay with them trying hard to keep bots out of my account.
The physical presence of banks is going away. Where I live you can't do any kind of monetary transaction in the local branch offices of any of the banks anymore. You can a) apply for a loan (and even that may go away soon), and b) identify yourself and get a physical token used for accessing the bank via the net. You can't withdraw money, you can't pay bills, you can't exchange currency. I haven't been inside my bank for many years, there's just nothing I can do there. The last time I visited the bank was with my wife (an immigrant), with her documents, to get her into the system.
Different customers have different attitudes towards this. Some of them are _very_ focused on conversion and will disable anything which causes additional user friction. For others, the economic damage of bots is just so painful that it makes economic sense for them to add friction for a few percent of users.
I'm a linux user myself, so I know for a fact that neither my previous employer, nor other bot vendors, will block linux user agents in particular. Customers generally don't mind a universal requirement for JS execution, so that's just a fact of life. We generally did try to avoid blocking privacy focused browsers, though. We certainly monitored false positive rates and knew pretty well how we affected users.
Cloudflare is pretty guilty of this if you use some more exotic approaches to request info. How often have I seen their captcha that is intended for bots...
>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech.
Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.
Okay let's go back to before I was born when people still used IRC and let's say you hated someone else's IRC server. You can just use a program to flood their server with garbage messages. In order to try and stop this spam they first try and deanonymize where this traffic is coming from. This can be done by looking at the IP that these bots are coming from. Now they can gline you and the flood ends. Now let's say the internet didn't leak your IP deanonymizing you. What are they to do? They essentially are forced to lock down the server and whitelist it. They can not allow anonymous users to join or else risk just being flooded.
Stopping abuse has always been a game of trying to deanonymize users in order to try and ban the harmful ones.
With many of these big anti-bot services like Google ReCaptcha, it's not even specialized anonymity tools that can cause shadow banning, just unusual user-agents.
All of these have independently caused me to get into endless ReCaptcha loops: firefox on android, smartphone with unusual screen resolution, clean browser profile with VPN.
It's so common that I now default to using duckduckgo, which never blocks me. I doubt DDG has a lower DDoS/Resources ratio than Google. Some companies are just lazier and less principled than others.
This is not quite up there with "won't someone think about the children!!!!", but still, it's sad.
Fortunately, almost all of the websites I visit with my anonymized browser aren't places that I wish to attempt to post a message. Unfortunately, I can easily run into defenses of an entire site when the problem is spam sending.
To kinda tweak this since people do tend to like their anonymity, "Do you have to be anonymous to all parties, all the time?"
Parent poster trusts his bank, and his bank would trust his once it knows he's not an fraudster, so maybe it's in everyone's interest to just allow the javascript for that one site.
Not to mention a lot of these bots are after scamming the company’s own customers. Breaking into accounts to commit fraudulent activity, to reach out and “recruit” people into whatever scam they are trying to run.
Nobody wants to spend time trying to stop these bots. It is, however, a very necessary thing to do.
I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.
Yeah, there’s a central service that all Flight search is connected to, irregardless of airline. The airlines are charged per search to that api, so they monitor their ”look to book” rationvery closely. That ratio remains quite stable im the absence of bots, but skyrockets with any bot activity. Hence, they know from that metric how big of a bot problem they have and how much money they are losing. Major flight search software vendors have dedicated teams for this.
In fact the airlines are charged per book, but if and only if the look to book stays within reasonable bounds. If it rockets up, they’re on the hook for the penalties
Saying that Anti-bot software is nonsense is like saying that door locks are snake oil too. We've all seen Lockpicking Lawyer on Youtube opening with ease any lock out there, so how come that all of us haven't got robbed yet?
Well, because protection is not a binary thing - either being 100% safe or 100% not working - instead it's a proportion between the skill/effort/time needed to break in, and the reward you get for it.
To stop majority of attacks you don't have to be absolutely unbreakable, you just need to make it hard enough for majority of attackers so that it doesn't payout for them compared to the value of the data you're protecting. And that's where anti-bot SW has it's place, it slows done spiders and global attacks, forcing for custom tailored scraping that is constantly being fine-tuned, infrastructure to hide your IPs, and that makes the operation way more expensive and harder to run continuously...
Back when we had to scrape airline websites to get the deals they withheld for themselves, residential IP was indeed the way. Once the cottoned on to it and blocked id, you'd simply cycle the ADSL model, get a new IP, and off you'd go again.
Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad allocated to web scraping). The airline threw ridiculous dollars at trying to stop it, and we just used a caffeine fueled nerd to keep it running. It wasn't all fun though, they'd often release their new anti scraping stuff on a Friday afternoon. They were less than impressed when they learnt who the 'enemy' was. Good times!
>Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.
Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?
Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?
Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser.
Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.
By default Selenium exposes a few things in JS that are pretty trivial to detect, so you need to disable/hide that for starters. I don't know how easy or hard that is, but stock Selenium is a poor way to get around anti-bot stuff.
Its fairly trivial, some stuff needs a modified browser executable or some JavaScript magic but it wont take you longer than 2-5 Hours to bypass most of the heuristics, disabling the window.navigator.webdriver flag and getting a residential IP is on its own enough to get single click captchas every 1-2 tries for example. To be fair I haven't looked into it since 2019 but i doubt that its gotten much harder.
Most flights are available through the airline booking systems such as Sabre. However, airlines might have flights available only on their own website at (sometimes massively) reduced cost, which needs to be booked through that site. So the web scraping became two parts, one to provide the data to our search engine to present to our customer (travel agent) customers. The second part was then we would book via the airlines website with the details provided by our customer's customer.
A residential IP would help for IP based detection. As the Readme mentions, there's also Javascript based detection. If, for example, your browser has navigator.webdriver set incorrectly, then you can still get blocked even on a residential IP.
The point is that both can be required. You can have the most sophisticated user emulating browser, bit if all you have access to run it on are low quality IPs that have been blocked or that are often used for abuse, you won't get far. You can have residential IPs and if you're just wrapping curl, you also might find you're blocked.
Together, there's little to detect different than a regular user. The reason why the residential IPs is given heavy importance though is that it's the one part that costs a lot of money if you need enough of them you need to use a proxy service and you transfer a lot of data. Entry level pricing is over $15/GB for high quality services.
This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs.
This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/
This is really bad. Imagine if someone plants these proxy inside app how user are even going to know? I think every OS should come with firewall so if app tries to make connection it should prompt with Accept | Accept Forever | Deny | Deny Forever.
I think these companies used to go for extension developer now it seems they have found new idea to implant malware on apps which is not easy to detect.
One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.
This is reason why most of services uses App instead of Browser. When there is App it can use lot of thing like phone fingerprint derived from various sources.
Note that Apple explicitly tries to prevent apps from generating any sort of overall device fingerprint besides the ad tracking identifier one which now requires user consent in iOS 14. You can still generate an app-scoped device ID though. (Not sure if these persist across re-installs of the same app or not)
There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.
Captcha solving services are a thing, can be as crude as something that takes a screenshot, sends an image to a click farm worker getting paid $300 a month sitting in a cubicle in Bangladesh.
There's various captcha solving services where you pay in bulk per captcha and submit data via an api.
Yup, such click farms exist. But driving up the costs and/or technical implementation efforts for bots/scrapers can be a part of your anti-bot strategy.
It's always just a question of detecting these things and code them in.
It's like writing a game bot with Java robots and pixel detection. It may is inefficient, may takes longer to make than a network solution. But I have yet to be detected anywhere
I've always thought credential stuffing and most password hacking attempts could be defeated by simply logging into randomly generated dummy accounts if the password is wrong. Just make it so that the same username / password combo takes you to the same random info. Real users should notice things were wrong immediately but bots would have no way to tell unless they already knew some of the real information.
> Anti-bot software is nonsense. Its snake oil sold to people without technical knowledge for heavy bucks.
I disagree. Obviously there is no way to 100% stop scraping, but a for a rather small amount of $ you can implement some measures that make it harder. Services like https://focsec.com/ offer ways to detect web scrapers using proxy/VPNs (one of the most common techniques) for little money.
> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.
Keep in mind that they may be legally or contractually forced to do this. Think of Netflix who are investing heavily into their Anti-VPN capabilities, most likely because they have contracts with content publishers & studios that force them to do so.
If users using weak/reused passwords is your problem, just don't let users choose a password (generate it for them), or don't use passwords at all (send link by e-mail that adds a cookie), or use oauth login.
Link-only login is the most underused security option, even more so for low-profile sites that need a minimal user account but do not really need full-on security.
I.e. what Facebook does if you don't log in for long enough. Two days ago I got a pair of messages to the same address with links to completely bypass login and verbiage about how they'd seen I was having trouble logging in followed an sms message with the same to a phone number they're not supposed to be using. It looks a lot like phishing, but it comes out of Facebook's servers and they've done it to me before.
I'm not the poster you're replying to, but: Facebook collects asks for your phone number for security/account recovery reasons, but then turns around and uses it to market to you.
Basically. Most websites already make you login with an email and verify you have access to that email and use the email as a password recovery mechanism. May as well just use the email itself as the login.
Of course if everyone did this, then all of your logins would have the same password (your email login).
> None of these anti-bot providers give a shit about invading your privacy, tracking your every movements, or whatever other power fantasy that can be imagined.
There is vast amount of profit available in doing just that (see e.g. GOOG and FB market cap). Even companies that truly have no intention of exploiting data collected as a side-effect of whatever product line they do, nearly always eventually end up going for that profit line. Because passing up on more income merely due to moral considerations is too much of a temptation for a company to be able to resist in the long term.
The credential stuffing wiki page didn't exist the last time I thought about invalid traffic so I'm pretty out of date.
How is there not an equilibrium here that cuts off credential stuffers? I'd naively imagine the residential IP providers have some measure of bad actors they themselves use to determine if a client is worth it, and that someone getting all your IPs blacklisted would get dropped pretty quickly.
In reality residential US ISPs don't really care if their users are getting a sub-par experience since they're often the only fast/fiber provider in the area of their customers, meaning customers have no way to switch. Plus, when a website doesn't work, unless the page itself calls out the ISP (which they never do), customers will think it's an issue with the website and won't possibly attribute blame to their ISP until they're deep in forum threads with people telling them "it's probably your ISP not doing anything about bad customers" - the amount of users going so far to learn this information, then accepting it, is extremely low.
On a site I used to run, there was no content which needed protection. So, it was not much of a pain except that there would be a lot of bot- filled contact forms. Slowly the problem became severe enough that bandwidth fees started becoming an issue. Finally had to use cloudflare in the front to reduce bandwidth usage. It worked but the side-effect was that some valid users may now get blocked.
I know how bad this issue is, and I wouldn't take down this repository. Anti-bot software does not work, anyone who pays 10m per year to have it simply has too much money.
If your password db is so broken that it's useful to create a term to abstract attacks ("credential stuffing"), then the right answer is to actually fix that security (eg pick users passwords for them, or completely replace with email auth), rather than thinking you're raising the bar by requiring attackers to come from a residential IP.
2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only.
2FA, especially app based, has been proven to work really really well.
It does not.
There are myriad ways of extracting the TOTP seed from these apps... Or you just reverse engineer the setup/confirmation process and then you can generate/trigger your own tokens from your automation workflow.
2FA is a good security feature but it does not help against web scraping. Credential stuffing and other 3rd party attacks? Yes, it _can_ help. But it does not always help. There's a phishing group that has seemingly specialised on getting people to click the green confirm button in their Duo app... ¯\_(ツ)_/¯
Check https://github.com/revalo/duo-bypass for a python script that can be used to automate Duo tokens... Has some code from me. There are similar scripts for all the other well known OTP Apps...
Having malware installed on every users phone is so many orders of magnitude harder than downloading the latest db dump and testing the email/password on every other site.
At the bare minimum, TFA stops most attacks. That's a whole lot better than the current situation.
There are different methods of 2FA like scanning encrypted barcodes that show that you require intent.
It seems that the Duo core app is a variant of HOTP?
What's the name of the phishing group and any details on them?
There was a Defcon or Black Hat video where they would constantly send a push approval to the mobile which was not PIN protected and most people would click on it. Don't remember which OTP generator it was.
How do you propose to implement two-factor authentication, on something like the public facing homepage of an airline ticket price search website, where if you make people "sign in with google" or whatever, a sizeable proportion won't do it and will just go to the competition?
thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)
2FA isn't limited to one device, or specific 2FA mobile apps. For example I use oathtool for most 2FA things; you just need to store the secret (often in the form of a QR code, but many services will also offer a text version, and if not you can decode the QR).
100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to contact them outside of it (I tried...) I need to switch banks.
Revolut now has a web app [1], which still tries to get you to log in via the mobile app but this is not necessary. So long as you know your pass code and have alternative access to your email then you can log in and do most of the things you can do via the app. You do have to wait 10 seconds for the privilege though (before allowing access via email there is a timer before you can confirm you do not have access to the mobile app.).
That sounds like a bad planning problem in which you shouldn't have left yourself vulnerable to loss of electronic services. Not a tech issue that justifies intrusive spyware.
I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?"
It sounds great but it is a completely ignorant thing to say.
When I worked in e-commerce as a SRE, bots were doing two things:
- trying to disrupt business processes (eg: false referral listings, gift card scams, etc)
- trying to disrupt systems
I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.
There's also a fair amount of scraping for things like...
- Reselling aggregated data
- Competitive pricing and inventory data
- "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve
- Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it.
- And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
> I'm sure there are folks who use bots and scrapers for home automation
I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.
I don't think anyone mentioned crawling (in my mind, "crawling" refers to a long-running process over several pages or sites and "scraping" could be a single fetch of a single URI)
I'm sure there are others, just from the top of my head:
* Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating.
* Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability
* Public transport: next bus/trains from closest station, delays and interruptions
* IMDB/tvdb/etc for monitored shows and movies. Common with sonarr.
Some people have smart mirrors or any other kind of 'news display', so it may be useful for them to scrape the data they may think relevant (this may be weather data, stocks, or even the new Nintendo Switch availability at their nearby retailer).
For whatever reason in years of running a SaaS this only happened maybe 3 times and never with my online shop. I guess using stripe and a few basic security settings keeps them away mostly
Nothing of course. But follow that string a little further.
So you do some thing which once a day scrapes a site and pulls off some data that you use in your thing. Maybe you talk about it to friends, or you have this thing as one of your github repositories. Some of your friends download the repository and also start using your thing. They talk to people about how cool your thing is, or what it does and the nice convenience of automating something that you used to have to do manually.
There are 86,400 seconds in a 24 hour period, probably folks won't change your code at all at first, and as it diffuses into the community some webmaster starts seeing this weird spike of queries that happen once a day at some time. Different addresses but always the same kind of request.
It's not a problem when its like 10 or 20 qps burst but when it starts getting up to 100 - 200 or worse 1000 - 2000, it causes the system to perhaps spin up additional instances that it isn't going to need after the burst and waste money. So the webmaster starts denying those requests with a 404.
Now sometimes your code works and sometimes it doesn't but you don't know that the webmaster is fighting you yet. Maybe eventually you start randomly varying the request time, or the people who have copied your thing are in more varied time zones so you it starts getting spread over the day.
now the webmaster is seeing bursts of traffic nearly every hour on the hour and that is weird so a more aggressive mitigation strategy is enacted.
People using your thing complain that it keeps breaking so you look into it and realize that the site is trying to block your requests. Perhaps you don't understand why this is, or perhaps you do and don't care, either way you come up with some strategies that avoid the block (maybe your rotate the user-agent or something).
Now the query traffic is spiking again and the webmaster is getting complaints that this 'bot traffic' is resulting in useless AWS fees because it isn't part of the revenue traffic and it is forcing the service to add more resources for their customers.
Not all scrapers are malicious, but my experience is that it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching) and because its all open source it spreads around.
> it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching)
That's exactly my case though. I have a few scraper scripts that I've never published. So what if it's rare? Do I deserve to be treated like a botnet just because it's inconvenient for some webmaster or company to do otherwise? That's not fair at all.
What I really enjoy about this thread is all of the completely different perspectives. Lots of people doing anti-abuse research bemoaning that this stuff exists, and lots of people working against what are from their perspective ham-handed anti-abuse tech blocking legitimate useful automation trading tips on how to do it better. I guess the other sides of those we don't see much. People doing actual black-hat work probably don't post about it on public forums, and most of the over-broad anti-abuse is probably a side effect of taking some anti-abuse tech and blindly applying it to the whole site just because that's simpler, often no tech people may be really involved at all.
When these companies endeavor to stop abuse, they trample all over our freedoms. Suddenly we can't have non-browser user agents anymore. Suddenly we can't root our smartphones anymore. They want nothing to do with us unless it's 100% on their terms with us completely under their control.
There seems to be wildly different perspectives on "bad actors means we can't have nice things" -- one group says that this is a fact of life, and the other says that this is an affront to freedom. A non-tech example is I've had guys on Tinder get legit angry at me for insisting that our first few dates have to be in public places where we drive separate -- "oh so you think I'm some creepy stalker?" And like I am totally empathetic to their hurt because I'm sure that they know they're good but I don't and there's now way for me to tell in advance. Malicious actors don't exactly announce themselves and actively try to hide their intent.
The solution to the automation problem is to do what most companies do and have registered API integrations.
> A non-tech example is I've had guys on Tinder get legit angry at me for insisting that our first few dates have to be in public places where we drive separate
That's totally understandable though. I insist on meeting in public as well... It's the real world, safety is most important.
The thing with websites is they already allow me to make thousands of requests from my browser. What harm does it do if I make a bunch of requests from a script? I don't see it.
> The solution to the automation problem is to do what most companies do and have registered API integrations.
Yeah, those are pretty great. I always use those whenever possible. Many of the sites I use lack those though. Some have APIs so badly designed that scraping their web site actually results in fewer requests and less overhead.
Agreed, I meant the original in a "this is why we can't have nice things" sense.
I generally appreciate registered API integrations, but the trouble is, for the most sites that are most problematic for benign automation, they usually don't have enough demand or revenue to justify well-maintained APIs.
I tend to think the solution is more to somehow make the market prefer more decentralized solutions, preferably federated. Not having one big target for bad actors means much less effort applied to attacking any one of the targets.
Dating might be a bit off topic, but I can see both sides as well. Women have genuine risks, and are very justified in taking precautions. But for the majority of decent guys, it can be tiresome to be constantly treated like you're an evil violent stalker. Maybe it needs a similar solution - a movement to local connections where people can have reputations that you can trust.
Yeah, but you're not entitled to use their servers. If your use of their servers is something they don't like, their freedom is to blackhole your packets
If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general.
Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.
Google can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access.
By the way great work on Marginalia search engine, I love it.
I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them.
A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in.
Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.
There's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.
>Legitimate uses of scraping include price comparison
"Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.
There are laws against unauthorized computer access.
This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.
For those that are interested in the specifics, Jamie Williams wrote a piece for the EFF[0] in the wake of hiQ vs Linkedin which dealt with this exact question.
It depends on who the server operator is. If it's your server, yeah, anyone I don't want to be there should go away. If it's your enemy's server, the argument that they're sending that page to the rest of the Internet turns out to be a decent one.
The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP.
Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
> Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response.
So it would seem that it actually doesn't positively imply that you're NOT authorized.
Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.
> The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource.
Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"
403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?
If you go by the semantics of what the 403 code means, absolutely, that's excatly what the status code means.
In practice there's of course nuance, like anyone will occasionally type in the wrong password on a log-in screen, maybe try again and then realize it was the wrong log-in prompt. That's mostly fine.
That's different from deliberate trying to circumvent a measure like this. If you are doing the stuff in the link, you are absolutely crossing a line and you know it.
There's a large difference between "I got a 403 so I hit F5 once" and "I got a 403 so I used a residential proxy and spoofed my user-agent".
> On the contrary, there are no laws that say you can't scrape a site.
You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.
In that case, the data compilation itself would be protected, not the individual data points. If I used a scraper to copy everything verbatim, then yes, it would be a violation.
I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices.
I haven't tried to scrape Ryanair, you could be right that it's trivial. It's the legal side that has a long and interesting history. Personally I wouldn't scrape them unless it was when working for a company that can afford lawyers.
> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that?
Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script?
Why is Google the only one who can do it? Must they have yet another monopoly?
> Bots with seconds between the requests rarely get blocked even by CDNs.
I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.
> Why is Google the only one who can do it? Must they have yet another monopoly?
I run a search engine and do my own crawling, and this does not correspond to my view of reality at all.
I have had almost no problems with getting blocked. If I have gotten blocked, it's usually been my own fault and I've been able to get unblocked by sending them an email explaining I run a search engine and asking for forgiveness because my bot wasn't behaving well.
The bots that do get blocked are bots in most cases bots that misbehave, ignore robots.txt, fetch the same resources repeatedly or with insane crawl-delays.
There are a few rare exceptions, but the whole "why does Google get a free pass when I don't?"-angle just doesn't hold water at all.
> I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.
Google doesn't repeat requests every day. I don't repeat requests every day. That's a weird thing to do, and it's well within a site owner's prerogative to block that nonsense.
> Google doesn't repeat requests every day. I don't repeat requests every day.
I scrape one site whose content changes every second. How is it "nonsense" to make one request every 24 hours? I make hundreds, thousands when I browse their site normally using my browser.
We've got people in this very thread talking about bots making hundreds of requests per second. How is one request every 24 hours harming anyone? People told me to make one request per hour to avoid hammering their servers, I decided to wait a day instead. It boggles my mind that this generous interval could possibly be considered abuse. How long should the interval be then? A month? A year? Infinitely long so the scraper never makes requests?
I built a search engine in school. I was lazy and stupid with the scraper and ended up writing a bug that caused it to loop on certain sites. That lead to the entire Duke law school site being DOS's during their class sign up period. Sorry for ruining scraping for everyone, but this is why websites don't want people scraping them.
But you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway?
> Why is Google the only one who can do it?
Because site operators explicitly allow them automated access. If you want the same treatment you have to ask for it.
> I can send you 403's just because I don't like your face.
Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.
Why do you think it’s dishonest to send 403s to bots but not browsers? Method of access matters — you the human might have access to your safety deposit box but the bank is still allowed to make rules about your access — like you have to come during business hours and you can’t send someone on your behalf.
I totally can though. If sign a document saying another person can do such and such on my behalf, that person can totally do that. Yes, even at the bank. No different from a user agent, really.
> Why is Google the only one who can do it? Must they have yet another monopoly?
Google can do it because most website operators want them to index their site. Plus, it is trivial to tell google to stop. That goes for all search engines.
That doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.
I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world.
In most cases you have very limited ability to decide what other people cannot do. And other people has mostly infinite choices of what they can do. I never heard anything as broad as you said. What you said is like a person standing on the street with a T-shirt says "do not look at me more than twice" and claim it has a legal binding to the whole world.
> I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world.
It kind of does though; if I own a store and say that only people with hats can enter, then I'm free to do so. Silly? Yes. Legal? Also yes.
There are some circumstances where it's not legal, mostly centred around discrimination. Details on this differ per jurisdiction, but generally speaking you have a right to refuse customers.
To me it seems sending a http request is somewhere inbetween looking (legal) and entering (illegal if not permitted).
However, most important is that the web as a system makes positive interactions easy and negative difficult. We have already found some set of constraints achieving this for interactions in public city streets. But it's not obvious the same rules (that we have internalized) have the same effect in another medium of communication.
If I can access the information in Chrome, I should be able to access the information with my own client, and the site operator shouldn't have any say in that.
> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.
Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.
Long-dos is usually not a botnet but a single or a few attackers requesting a resource heavy page on and on or using a technic like slowloris. If you are using WordPress I am sure you find some signs in your log.
The solution other than with ddos is to make the app better. Caching, access control and limiting, etc.
I'm a bot, you're a bot, we are all bots, so what does this even mean.
If someone is trying to discriminate between one set of visitors that they welcome and another that they deny service to, by definition they are the ones creating an adversarial environment. They are going to have to work to get the terms they want. Asking the players on the opposite side of the table to comply with their imagined rules out of altruism is a non-starter.
Technically it should be illegal to scrape websites without the consent of the server owner because it would be a violation of their property rights. That being said, I think that in reality it would be pretty difficult to get courts to agree with you and then to enforce it.
I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.
I have a considerable amount of experience in the industry.
Some of these so-called "advanced" techniques:
* We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below).
* We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
* All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this.
* With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers.
* We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software.
* Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).
This list is pretty interesting. If you don't mind me asking, what do you work on that you requires such sophisticated stuff?
Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct?
Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?
Mobile IPs will be a problem until the entire Internet is IPv6. The issue is that there are not enough IPv4 addresses for everyone to get their own IP every time their phone connects to the Internet. So the mobile networks use one IP for many handsets. Block the IP, block dozens of different (innocent) people.
Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.
Doesn't the same idea apply for normal IPs (non-mobile IPs)? Our computers also don't get their separate IPv4 address. We use NATs. So how can someone block our computer specifically without blocking dozens of innocent people using the same public IPv4 address?
Basically what's different about mobile and laptop IPs?
CGNAT is done at a ridiculously large scale in mobile networks.
Your home or local coffee shop might have 10-20 users max behind a single IP. A mobile network on the other hand might put most a small-medium city behind 3 or 4 IPs.
Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.
Not the OP but when I was running Operations at Blekko (a search engine) I spent part of my time dealing with scrapers.
When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits.
1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit.
2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing.
3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator.
One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.
It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.
> One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.
Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.
Well if you missed it, back in the day, places like Download.com or Source Forge would have a little pre-clicked link to "add <name>search toolbar to my browser!" which, after you downloaded your think would leave you with a browser that always redirected your search requests to what ever company paid to send you there.
Lots of people did it, even Blekko (although we stopped after we figured out it was just a scam), and the way it worked is toolbar company X would approach your internet site and say "we can send you a lot of traffic, just use this toolbar, we'll even pay you every time one gets installed."
Anyway, the toolbar would hook itself into the address bar and search selection hook in the browser and redirect every search to where ever it was told to. The nefarious part is that these tool bars were often shipped to the company as binaries not source, so you didn't really know everything they were doing. Once we figured out they were scumbags we also found out that they did some really scumbaggish things.
We stopped using them but lots of people did and finally the browser makers re-wrote the browsers to make what they did either impossible or easy for the user to revert and those guys rolled up their shops and went on to become some other type of scam.
Abuse is the kind of problem area where anyone seriously working on either side will be unlikely to go into the details. It's the same whether it's blocking scraping, stopping spam, preventing account takeovers, or detecting payment fraud.
In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff.
It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows.
In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.
Sorry about that, we're unable to discuss our projects with adjacent teams within the company. What you're saying is a valid frustration, the motivation behind the original comment was to put a thermometer on the repo.
Adjacent teams? It makes sense not to discuss it with totally unrelated teams, but a large Internet company wants to have its cake and eat it too - scrape all of the worlds websites, while simultaneously denying others from scraping its results. Limiting communication between adjacent teams (ie the scraper team and the bot blocker team) really seems like it would be a hinder rather than help.
There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blurb for each search result.
This kind of indirect scraping can be useful for getting almost all the information you want from sites like LinkedIn that do aggressive scraping detection.
I ended up having to put my search engine behind a CDN and CAPTCHA every new IP because I got something like 30,000 search requests like this _per_hour_ from a botnet. They seem to have backed off now but they were really quite persistent for a while.
almost all sites uses "host" command to see if the IP belongs to the googlebot (this is recommended way by Google). And you can easily find many IPs which return googlebot when you "host" them. So you can use any of these IP addresses to spoof GoogleBot.
To find more IPs, make your own website, and wait until GoogleBot eventually shows up :)
It's very easy to install Chrome on a linux box and launch it with a whitelisted extension. You can run Xorg using the dummy driver and get a full Chrome instance (i.e. not headless). You can even enable the DevTools API programmatically. I don't see how this would be detectable, and probably a lot safer than downloading a random browser package from an unknown developer.
Hmm maybe I will if I have time. We've been using this technique for user-initiated scraping. The only issue we've run in to is we get rate-limited by IP sometimes. Changing the IP has solved the problem each time.
If I am correct in assuming the parent is talking about puppeteer, there is a plugin[1] that claims to evade most of the methods used to detect headless browsers. I have used it recently for just that purpose, and I can say that it worked wifh minimal setup and configuration for my usecase, but I guess depending on the detection mechanisms youre evading YMMV.
The creator of that plugin does mention it is very much a cat and mouse game, just like most of the “scraping industry”
Google "residential proxies for sale" if you want to see the weird shady grey market for proxies when you need your traffic to come from things like cablemodem operator ASNs' DHCP pools
Wonder what fraction of that traffic is from p0wned IoT refrigerators, smoke detectors or WiFi enabled light bulbs… probably more than anybody cares to admit…
Some amount, probably, but some of these "residential IP" providers just buy IP blocks from small residential or business ISPs. (Or even less shadily, buy IP transit from them and get assigned a block of IPs that the databases think are normal residential/business users.)
The majority of them want to do something completely benign like see a BBC show in the US, or watch an American football show in the UK, and the one defining feature of capitalism is its many contradictory faces.
One corporation wants to arbitrarily limit its customer base, and the other corporation wants to arbitrarily limit what data its customer base can see. In between the two is a space big enough to drive a Mack truck or a lorry through, depending on where you're from...
One might justify this until the one corporation merges with the other and then you have a situation where the same corporation wants to do two different things to the same pool of users.
When I say tricked, I mean they have no idea that unknown 3rd parties' grey market traffic is being run through their personal home internet connection. They believe they are just using a VPN to watch BBC.
Yes, but my point is they've not been duped as much by the "free VPN" provider as much as they've been duped by the people who created the market for the "free VPN" provider.
I found out while sorting through business contributors to a non-profit once that a pretty big market for mobile relays is the "free VPN" offered on the app stores to high school kids looking to circumvent the outgoing blocks on the school's wifi.
In that case the school's administration could easily purge the "free VPN" of local users by removing the wifi restrictions. Instead, they serve more traffic to more nefarious places to maintain an illusion of control not for the kids in the school, but for themselves.
All of this is basically Dr. Strangelove but with spyware rather than nuclear bombs.
Having scraped million of webpages, I find dynamic CSS selectors a bigger time sink than most anti-scraping tech encountered so far (if your goal is to extract structured data).
Can your scraper be used to scrape images? I need to scrape some books from a paywalled site and they are presented a page at a time. The JS code is too complex for me to bother trying to figure out how it creates the unique tokens it applies to every image it displays to avoid a very simple scrape.
2 of my social media accounts have fallen victim to bot detection, despite not using scripts. There are other websites for which I have used scripts, and sometimes ran into CAPTCHA restrictions, but was able to adjust the rate to stay within limits.
CouchSurfing blocked me after I manually searched for the number of active hosts in each country (191 searches), and posted the results on Facebook. Basically I questioned their claim that they have 15 million users - although that may be their total number of registered accounts, the real number of users is about 350k. They didn't like that I said that (on Facebook) so they banned my CouchSurfing account. They refused to give a reason, but it was a month after gathering the data, so I know that it was retaliation for publication.
LinkedIn blocked me 10 days ago, and I'm still trying to appeal to get my account back.
A colleague was leaving, and his manager asked me to ask people around the company to sign his leaving card. Rather than go to 197 people directly, I intentionally wanted to target those who could also help with the software language translation project (my actual work). So I read the list of names, cut it down to 70 "international" people, and started searching for their names on Google. Then I clicked on the first result, usually LinkedIn or Facebook.
The data was useful, and I was able to find willing volunteers for Malay, Russian, and Brazilian Portuguese!
After finding the languages from 55 colleagues over 2 hours, LinkedIn asked for an identity verification: upload a photo of my passport. No problem, I uploaded it. I also sent them a full explanation of what I was doing, why, how it was useful, and a proof of my Google search history.
But rather than reactivate my account, LinkedIn have permanently banned me, and will not explain why.
"We appreciate the time and effort behind your response to us. However, LinkedIn has reviewed your request to appeal the restriction placed on your account and will be maintaining our original decision. This means that access to the account will remain restricted.
We are not at liberty to share any details around investigations, or interpret the terms of service for you."
So when the CAPTCHA says "Are you a robot?", I'm really not sure. Like Pinocchio, "I'm a real boy!"
CouchSurfing is just shit, full stop. I love the concept and hosted many people, but the way the company has been run over the last few years is beyond atrocious. It's like AirBnB sent over some people to intentionally run it in to the ground or something.
LinkedIn has to deal with a lot of scummy recruiters and scammers; I don't blame them for being very strict.
I knew there was a reason why I used client certificates and alternate ports.
Why is it so difficult to just respect robots.txt? Maybe there's an idea for a browser plugin that determines if you can easily scrape the data or not. If not, then the website is blocked and then traffic will drop. I know this is a naive idea...
Never underestimate the scraping technique of last resort: paying people on Mechanical Turk or equivalent to browse to the site and get the data you want
Are there any court cases that provide precedence regarding the legality of web scraping?
I'm currently looking for ways to get real estate listings in a particular area and apparently the only real solution is the scrape the few big online listing sites.
I think it only applies to systems that aren't available to the general public, which in this case was the GCIC. Anything that is available to the public, even if it requires some sort of registration, would I think be legal to scrape. YMMV though.
I was involved in a scraping-related case, though in my situation we were scraping public domain data/facts/public domain media. Email me if you'd like additional info. :)
More related to the submission content -- at the time we used rotating proxies, both in-house & external (ProxyMesh - still exists & only good things to say about it); they allowed us to "pin" multiple requests to an IP or to fetch a new IP, etc...
It’s most likely for tracking clicks. Better to just search for the company names instead of clicking on the links in case they lead to unexpected places.
It always amazes me how people believe they have a right to retrive data from a website. The HTTP protocol calls it a request for a reason: you are asking for data. The server is allowed to say no, for any reason it likes, even a reason you don't agree with.
This whole field of scraping and anti-bot technology is an arms race: one side gets better at something, the other side gets better at countering it. An arms race benefits no one but the arms dealers.
If we translate this behavior into the real world, it ends up looking like https://xkcd.com/1499
Because often that data is only available through scraping.
Nobody wants to scrape, it's messy and fickle and a general pain in the backside. But sometimes the data you need exists only in that form.
If you run a website and you have a problem with scrapers, then make all that data available through an API and say what acceptable rate limits are. If cost is an issue, then charge a proportionate fee, my time writing a scraper is worth much more than paying a few dollars for an API.
If you just say "No" to everything then you lose all control over the process and the only outcome will be such an arm race.
God. This. The number of times I've spent 2 days of my very expensive time coding a scraper to get data I'll use once, when I would have paid a few dollars just to download it in a text file.
For the row "Long-lived sessions after sign-in" the author mentions that this solution is for social media automation i.e. you build a tool to automate social media accounts to manage ads more efficiently.
I am curious by what the author means by automating social media accounts to manage ads more efficiently
Trying to stop credential stuffing by blocking bots will not work, and can often severely impact people depending on assistive technologies.
I think a better solution is to implement 2FA/MFA (even bad 2FA/MFA like SMS or email will block the mass attacks, for people worried about targeted attacks let them use a token or software token app) or SSO (e.g. sign in with Google/Microsoft/Facebook/Linkedin/Twitter who can generally do a better job securing accounts than some random website). SSO is also a lot less hassle in the long term that 2FA/MFA for most users (major note: public use computers, but that's a tough problem to solve security wise, no matter what).
Better account security is, well, better, regardless of the bot/credential stuffing/etc problem.
A lot of web scraping is annoying often because there’s *an explicit API built for the scrapers needs*. Instead of looking for an API, many think to first use web scraping. This in turn puts load and complexity on the user facing web app that must now tell scraper from real users.
Using the API almost always has more "strings attached". Like you have to register and get an API token or something. Or even pay. If you want people to use your API, don't make it less convinient than scraping the page.
But if there's an API, then the overall load is the same, no?
Or to put it another way, naively, having api.example.com and realpeople.example.com separated out into separate sandboxes seems reasonable, but due to the aforementioned problem, its not. But then it also turns out to be the wrong axis for this anyway, and you need your monitoring to work for you.
No, the load isn't the same because the web page might be a multi-megabyte monster piece of badly-coded HTML that returns only say 10 out of 1,000,000 results and needs to be paged through, where the API might return all the million results in a nice JSON chunk.
I am running a no-code web automation and data extraction tool called https://automatio.co. And from my experience most of the time when using quality residential proxies you will be fine. But that comes at cost since they are way expensive then data center proxies.
But for some websites, even residential ips doesn't let you pass.
I noticed there is like a premium reCaptcha service, which just work differently then standard one and not let you pass. It's mostly shown with a Cloud flare anti bot page.
By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients.
I know you can put a meta tag, but bot still has to make a request to read it.
I would like to completely cut off Google from scrapping.
You can buy databases of who owns which IP blocks.
If you really care but don't want to spend the money, just block the subnet each time you see a Googlebot request. "whois w.x.y.z" returns an entire CIDR, and it seems unlikely to me that Google is scraping from a bunch of disconnected /24s.
I’ve tried skirting datadome but generally you can just get around it by rotating ips, apparently there is a way to de-obfuscate their apps (apps that use datadome services) to retrieve datadome cookies but I haven’t been bothered to check it out yet.
In a previous venture my team successfully circumvented bot detection for a price comparison project simply by using apify.com. Wasn't that expensive, either. We were drilling sites with 500k+ hits per day for months.
If this guy got to experience how systemically bad the credential stuffing problem is, he'd probably take down the whole repository.
None of these anti-bot providers give a shit about invading your privacy, tracking your every movements, or whatever other power fantasy that can be imagined. Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.