Google On Authentication For SEO Tools & Personal Crawlers – Search Engine Roundtable
Google’s John Mueller posted saying, “Authentication for your personal crawlers & SEO tools is going to be more and more of a topic.” I believe, but I can be wrong, this is alluding to the num parameter being disabled and how tools are having issues now with scraping Google.
He wrote on Bluesky:
Authentication for your personal crawlers & SEO tools is going to be more and more of a topic. This is a simple guide to get started (though I imagine it’ll evolve significantly over time).
The neat thing is that you don’t have to list your IP addresses, and can be verified from where-ever you are (provided you keep the signature). Ideally you’d authenticate on more than just the signature, so that people can’t fake you, but this is early steps.
If you’d like to be involved in the web standards around this, follow datatracker.ietf.org. and the associated mailing list. Your feedback on all of this is welcome there, no need to be an “expert” (whatever that is).
An example of something that’s still uncertain: if you have an AI system in your browser that clicks around and does stuff for you, is that a bot that could (should?) be authenticated? or is that just “you”? (a better / different / assisted “you”?)
I mean, is there a way these third party tools can authenticate themselves so they can go beyond 20 results? I suspect not?
Forum discussion at Bluesky.
Note: This was pre-written and scheduled to be posted today, I am currently offline for Rosh Hashanah.
The content at the Search Engine Roundtable are the sole opinion of the authors and in no way reflect views of RustyBrick ®, Inc
Copyright © 1994-2025 RustyBrick ®, Inc. Web Development All Rights Reserved.
This work by Search Engine Roundtable is licensed under a Creative Commons Attribution 3.0 United States License. Creative Commons License and YouTube videos under YouTube’s ToS.
