New research is pushing back hard against the idea that “AI visibility,” tracking how often a brand shows up in ChatGPT or Gemini answers, deserves the budget marketing teams have started throwing at it, and Search Engine Roundtable’s rundown of the latest studies lands at an awkward time for an entire new category of SEO tools built around measuring exactly that.
Over the past year, a wave of startups and established SEO platforms alike have launched “AI visibility” or “AI search optimization” dashboards, promising to tell marketers how often their brand gets mentioned when someone asks ChatGPT, Gemini or Perplexity a relevant question. The pitch has been straightforward: traditional keyword rankings matter less as more search behavior moves into conversational AI tools, so you need a new metric and a new dashboard to track your standing in that world. Agencies have built entire new service lines around it. Budget has followed.
The studies Search Engine Roundtable rounded up this week argue that pitch is running well ahead of the evidence. The core problem researchers keep bumping into is measurement instability: ask the same AI assistant the same question twice, minutes apart, and you can get different brand mentions, different ordering, sometimes different answers entirely. That’s a fundamentally different problem than traditional rank tracking, where a given keyword’s position on a given day in a given location is at least a stable, repeatable number you can plot over time. If the thing you’re measuring changes every time you measure it, a trend line built from repeated measurements is closer to noise than signal.
There’s a second, subtler issue the research raises: even when a brand does get mentioned in an AI answer, there’s limited evidence connecting that mention to any business outcome marketers actually care about, click-throughs, conversions, brand recall. Traditional SEO at least has decades of data correlating ranking position with traffic. AI visibility tracking is trying to establish that same kind of correlation in real time, with a product category barely two years old and platforms that change their underlying models and retrieval behavior on a schedule nobody outside the labs controls.
“We’re measuring something real, brands genuinely do appear differently across AI answers, but we’re measuring it with tools that weren’t built to handle this much variance,” one researcher cited in the roundup said. “Right now a lot of ‘AI visibility scores’ are precise-looking numbers built on an unstable foundation.”
None of this means AI visibility is a fake problem or that brands should ignore how they show up in AI-generated answers. It means the current generation of tools measuring it are earlier-stage and less reliable than the marketing around them suggests, and spending heavily to optimize against a metric that swings wildly between measurements is a real risk of wasted budget. The more defensible version of this work, according to several of the practitioners quoted, is treating AI visibility tracking as a qualitative signal, worth checking periodically to understand general sentiment and accuracy of how an AI model describes your brand, rather than a quantitative KPI you report to leadership every month with confidence intervals attached.
That distinction matters because budget decisions are already being made on the more confident version of the story. Agencies pitching “AI SEO” retainers to clients have, in plenty of cases Search Engine Roundtable and others have documented this year, presented AI visibility scores with the same precision and authority as a Google ranking position, without disclosing how much those scores can shift between two queries run an hour apart. Clients paying for that work deserve to know the difference.
This skepticism lands alongside Google’s own messaging push this year, which has leaned hard into telling publishers and SEOs that classic fundamentals, content quality, technical crawlability, genuine expertise, still matter more than chasing AI-specific optimization tricks. Google has an obvious incentive to say that, since it benefits from SEOs not overcorrecting away from the signals Google’s own search ranking still weighs heavily. But in this specific case, the incentive and the evidence happen to point the same direction: the fundamentals that made a page rankable in traditional search are substantially the same fundamentals that make a page more likely to get cited accurately by an AI model summarizing the web.
For marketing teams deciding what to do with this information right now, the practical read is this: don’t rip out an AI visibility tool you’re already using, but stop treating its output as a precise metric worth obsessing over week to week. Spot-check it monthly, watch for genuinely alarming patterns like your brand being consistently misdescribed or associated with a competitor’s product, and keep the bulk of your SEO investment where the evidence is still strongest, which remains traditional technical and content fundamentals rather than a new discipline still working out basic measurement reliability.
It’s also worth noting who’s funding and publishing some of this skeptical research, since that context matters for how much weight to give it. A chunk of the pushback is coming from established SEO platforms that built their businesses on traditional rank tracking and have an obvious commercial interest in casting doubt on a newer category of tool that could eventually replace part of what they sell. That doesn’t make the underlying measurement critique wrong, the variance problem researchers are describing is real and easy to reproduce yourself by just asking an AI assistant the same brand question twice in a row, but it’s a reason to weigh the “AI visibility is overhyped” argument with the same skepticism you’d apply to the “AI visibility is essential” argument coming from the vendors selling the dashboards.
The bottom line: AI visibility is a real thing worth knowing about, but the tools measuring it right now are precise-looking numbers sitting on a shaky, high-variance foundation. Treat your AI visibility score the way you’d treat an unverified rumor, worth a glance, not worth a strategy pivot.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
