Neil Emeigh, the CEO of Rayobyte, answers some key questions about the changing landscape of web scraping and ethical data acquisition.
How do you envision the role of ethical practices in the future of proxies and web scraping?
In the proxy industry, it’s all too common for companies to simply embed consent somewhere deep within their Terms & Conditions and consider their responsibility fulfilled. Regrettably, many residential proxy network participants are unaware that their IP addresses are being utilised, a practice which I’ve always found to be unsettling.
Looking ahead, I believe businesses worldwide will become increasingly conscientious, scrutinising the origins of IPs and their acquisition methods before finalising agreements with providers.
What are some of the most notable advancements in web scraping technology and methodologies that you’ve observed in recent years, and how do you see these advancements shaping the future of data acquisition?
In the eight years since I founded my company, there have been significant shifts in the web scraping and proxy industries. Some of the most notable changes include increased difficulty in scraping websites.
When I first launched the company, our primary focus was serving SERP scraping clients. While back then, data centre proxies were the preferred method, it’s now nearly impossible to scrape Google at scale using them, and it has become prohibitively expensive for the average business owner.
To navigate these challenges, we and many of our clients have enhanced our anti-bot evasion measures. This includes improving fingerprint technology, developing advanced browser-based automation, and transitioning from data centre proxies to more sophisticated residential proxies.
With advancements like ChatGPT, integrating Artificial Intelligence (AI) into web scraping has become more accessible. Even regular developers can now leverage AI in their scraping processes.
More frameworks, libraries, and no-code solutions are making it easier than ever for developers (and non-developers!) to be able to scrape – which wasn’t true eight years ago. Now a data scientist with basic Python experience can build a full-fledged, scalable, web scraper himself.
With the increasing importance of alternative data for businesses and decision-making, what industries or sectors do you believe will benefit the most from web scraping and alternative data usage in the coming years?
Instead of singling out specific sectors, I’d propose a broader perspective: which industries and businesses wouldn’t gain an advantage from web scraping? Its applications have expanded far beyond just securing limited-edition sneakers or monitoring prices. It’s challenging to identify any sector that wouldn’t derive value from the actionable insights harvested from web-scraped data.
Our main challenge moving forward is crafting an ethical framework. By doing so, we can encourage more companies to confidently tap into the vast potential of alternative data. If we navigate this collectively, the pertinent question shifts from who will benefit to who won’t.
How do you see the relationship between web scraping and AI evolving in the future?
As an example, at Rayobyte, we handle billions of requests monthly. Given this immense scale, to make sense of this data and to use it to our customer’s advantage, we must tap into automation and AI. Our approach focuses on evaluating traffic patterns entering our system, with two primary objectives.
Automatic abuse prevention: Abuse poses one of the most significant challenges for proxy providers. Mishandled, especially within a residential network, it can wreak considerable havoc.
We integrate a combination of predefined rules, triggers, and AI to discern abusive traffic. A basic, albeit illustrative example would be monitoring HTTP 200 codes on target websites. If there’s an unexpected surge in request rates coupled with the emergence of non-200 HTTP codes, it’s a strong indicator of a potential DDoS attack. While such parameters can be manually set by our abuse team, leveraging AI’s anomaly detection capabilities can help us uncover more such patterns.
Intelligent routing: Within a rotating pool of proxies, including residential, ISP, and data centre, the foremost responsibility of a proxy provider is ensuring the proxy’s functionality, meaning that it’s accessible and hasn’t been banned. Drawing from historical performance data of IPs, a provider can discern the ideal one for traffic routing. With billions of requests each month, this vast dataset offers a fertile ground for training Machine Learning (ML) algorithms. These algorithms, when trained well, can optimise routing based on success rates rather than sticking to conventional static methods like round-robin routing.
From the perspective of a proxy provider, these are our key strategies. However, the intrigue amplifies when one dives into the domain of web scraping. Given the continuous updates websites make to deter evasions, it becomes inefficient to manually oversee configurations at a large scale. ML’s ability to learn, adapt, and modify configurations in real-time allows a scraping firm to remain one step ahead of evolving web dynamics.
