I'm not fully convinced scraping has actually gotten harder.. It feels more like the average approach has gotten softer.

Lately everything gets framed as rising costs or unstoppable anti-bot systems, but most sites didn't suddenly become impenetrable. What changed is how people react to friction.

We're in an AI-autopilot phase now. Hit a block and the instinct is to buy more credits, switch vendors,, or let an API abstract the problem away. Meanwhile, teams still doing basic engineering work around sessions, behavior, pacing, and retries are often scraping the same targets just fine.

Honest question: have scraping costs really exploded, or have engineering standards quietly dropped as abstraction layers piled up?

Interesting take on it. Some people probably wouldn't like to be called soft but there is likely some truth to it.

I feel it really comes down to priorities.

Scraping has always been a means to a end for most companies. Get data and then use it for something valuable. Before getting the data was easy, but now it is getting increasingly harder.

I think the key here is highlighting the fact that the time of cheap/easy/low skilled access to web data is ending. Companies either need to skill up on understanding how to bypass anti-bots or pay someone else to do it for them and they focus on the data.

I just worry we're collapsing two things into one bucket: harder in absolute terms vs harder relative to how much real engineering effort teams are willing to invest.

Those aren't the same, and to me the distinction matters.

Ok, I share my screen all the time.. not going to happen

The basic question is how much context do you actually want if you can't really affect it and it's therefore more of a distraction than useful input. Some is almost certainly useful but it probably varies by individual and situation.