website SEOWebsite Traffic
Download your cheat sheet and checklist to start building content that works harder.
Large AI Overviews on SERPs are affecting visibility and causing a dramatic decrease in traffic.
Join us for a data-backed session where we break down how to detect, diagnose, and eliminate unnecessary branded ad spend.
Large AI Overviews on SERPs are affecting visibility and causing a dramatic decrease in traffic.
This template is your no-nonsense roadmap to a flexible, agile social media strategy.
Large AI Overviews on SERPs are affecting visibility and causing a dramatic decrease in traffic.
Google is blocking search result scraping, causing global outages at rank and keyword tracking services
Google cracked down on web scrapers that harvest search results data, triggering global outages at many popular rank tracking tools like Semrush that depend on providing fresh data from search results pages.
What happens if Google’s SERPs are completely blocked? A certain amount of data provided by tracking services have long been extrapolated by algorithms from a variety of data sources. It’s possible that one way around the current block is to extrapolate the data from other sources.
What Google is doing is still having a major effect on the freshness of data that SEO tools are able to deliver. There are many SEO tools that are experiencing outages to data that can normally be obtained through scraping Google’s search engine results pages (SERPs).
@RyanJones, who operates several tools, tweeted an update today:
“Definitely affecting my tools as well – as we use a 3rd party data supplier and ALL the major ones were blocked yesterday. Many still are”
@seovision tweeted observations in Spanish (translated below) using a Spanish colloquialism to describe Google as a dog guarding vegetables it won’t eat, blocking the gardener who wants them and leaving the resource inaccessible to everyone.
“Since yesterday it seems that they have put in place a new anti-scraping system also in SERPs, which is stricter. They are getting very tough on scraping. …Like the gardener’s dog, I won’t sell you the data or let you get it.”
Semrush is likely the most used tool whose data has not been refreshed (according to posts online). That said, someone from Semrush contacted me to say that nothing unusual is going on and that there are no widespread issues.
Popular SEO tool, SE Ranking, is another service that’s experiencing a loss of fresh data.
@LauraChiocciora posted a screenshot of a message received from the SE Ranking tool indicating that position tracking is back online but that SERP Features is still missing because of “technical issues.”
The full message in the screenshot is:
“Position tracking is back online. SERP Features tracking is still not available due to technical issues. Our team is already working on resolving the problem and providing you with the data as soon as possible.”
Google’s guidelines have long prohibited automated rank checking in the search results but apparently Google has also allowed many companies to scrape their search results and charge for accessing ranking data for the purposes of tracking keywords and rankings.
According to Google’s guidelines:
“Machine-generated traffic (also called automated traffic) refers to the practice of sending automated queries to Google. This includes scraping results for rank-checking purposes or other types of automated access to Google Search conducted without express permission. Machine-generated traffic consumes resources and interferes with our ability to best serve users. Such activities violate our spam policies and the Google Terms of Service.”
Related: 13 Black Hat Techniques That Can Harm An SEO Campaign
It’s highly resource intensive to block scrapers, especially because they can respond to blocks by doing things like changing their IP address and user agent to get by any blocks. Another way to block scrapers is through targeting specific behaviors like how many pages are requested by a user. Excessive amounts of page requests can trigger a block. The problem to that approach is that it can become resource intensive keeping track of all the blocked IP addresses which can quickly number in the millions.
A post in the private SEO Signals Lab Facebook Group announced that Google was striking hard against web scrapers, with one member commenting that the Scrape Owl tool wasn’t working for them while others cited that Semrush’s data has not updated.
Another post, this time on LinkedIn, noted multiple tools that weren’t refreshing their content but it also noted that the blocking hasn’t affected all data providers, noting that Sistrix and MonitorRank were still working. Someone from a company called HaloScan reported that they made adjustments to resume scraping data from Google and have recovered and someone else reported that another tool called MyRankingMetrics is still reporting data.
So whatever Google is doing it’s not currently affecting all scrapers. It may be that Google is targeting certain scraping behavior, learning from the responses and improving their blocking ability. The coming weeks may reveal that Google is improving its ability to block scrapers or it’s only targeting the biggest ones.
Another post on LinkedIn provided insights about the impact, including speculating how the blocking could eventually result in higher resources and fees charged to end users of SaaS SEO tools.
Natalia Witczyk (LinkedIn profile) offered the following insights via LinkedIn:
“Fresh in: Google starts intensifying its anti-scraping measures, introducing stricter protections such as IP blocking and CAPTCHAs. Popular SEO tools like Semrush, SE Ranking are being impacted.
This move from Google is making data extraction more challenging and costly. As a result, users may face higher subscription fees.
Any of you seeing data issues in your SEO tools?
EDIT: Ahrefs claims no outages, so I have removed their name from the first paragraph, however some users reported data lags. Sistrix seem unaffected.”
Ryan Jones tweeted:
“Google seems to have made an update last night that blocks most scrapers and many APIs.
Google, just give us a paid API for search results. we’ll pay you instead.”
So far there has not been any announcement by Google but it may be that the chatter online may force someone at Google to consider making a statement.
Featured Image by Shutterstock/Krakenimages.com
I have 25 years hands-on experience in SEO, evolving along with the search engines by keeping up with the latest …
Join 75,000+ Digital Leaders.
Learn how to connect search, AI, and PPC into one unstoppable strategy.
Join 75,000+ Digital Leaders.
Learn how to connect search, AI, and PPC into one unstoppable strategy.
Join 75,000+ Digital Leaders.
Learn how to connect search, AI, and PPC into one unstoppable strategy.
Join 75,000+ Digital Leaders.
Learn how to connect search, AI, and PPC into one unstoppable strategy.
In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses — and careers.
Copyright © 2025 Search Engine Journal. All rights reserved. Published by Alpha Brand Media.