Why AEO and GEO Ranking Tools Cannot Be Objectively Accurate

The problem is not that a vendor is trying badly

AEO and GEO platforms are responding to a real need. Teams want to know whether ChatGPT, Google AI Overviews, Perplexity, Gemini, and other answer products mention their brand, represent it correctly, and cite their sources. Tools such as Profound, Brandlight, Peec AI, and Otterly.AI can make repeated observation much easier than asking every question by hand.

The technical objection is narrower and more important: no third-party tool can report one objectively accurate, product-wide “AI ranking” for a brand. There is no universal results page, fixed position, shared index, or published cross-product scoring function waiting to be read. There is an experiment, defined by a tool’s prompts, accounts, locations, model choices, retrieval path, timing, parser, and scoring rubric.

Start with the estimand: what exactly is being measured?

In measurement language, an estimand is the quantity a study intends to estimate. “Our AI rank” is not an estimand because it does not specify the population of questions, the answer products, the market, the time window, or the unit being counted. A defensible alternative is narrower: “the proportion of answers in version 3 of our US English comparison-question panel that mentioned the brand or an owned product, under the recorded run conditions.”

That sentence is less exciting than a score from 0 to 100, but it is testable. It also prevents a common category error: treating visibility in a sampled answer panel as if it were market share, customer awareness, or a stable search-engine position.

1. Generation is stochastic, and answers are not database rows

A generated answer is a sampled output from a model and a context, not a permanent row with one value. Small changes in decoding, system instructions, conversation history, tool results, or available sources can change wording, selected entities, and citations. Even when a product tries to make an experience consistent, a measurement system should not assume that one observed response exhausts the possible responses.

This does not require claiming that every product exposes a particular temperature setting or that a named vendor uses randomness in a specific way. The conservative point is that an answer is an output of a probabilistic generation pipeline, and the externally visible response can vary. If a tool takes one run as the truth, it is measuring one draw, not the distribution.

Repeated runs turn an apparent rank into an observed rate. A brand that appears in 8 of 10 runs is not necessarily “rank 1”; it is a brand with an 80% observed presence in that test under those conditions. The difference matters when the next ten runs produce a different result.

2. One prompt can become many searches

Search-enabled answer products may rewrite a user’s request into one or more targeted queries. OpenAI documents this behavior for ChatGPT Search and gives location-aware examples; Google documents grounding and supporting web content in its AI experiences; Google’s Gemini documentation exposes queries and web chunks in grounding metadata. The visible answer therefore reflects not only the text entered by the researcher, but also hidden query planning and retrieval.

This query fan-out creates two sampling layers. The measurement tool samples the prompt. The answer product may sample or generate the retrieval queries. A tool can faithfully replay the same prompt while still receiving different candidate evidence after an index update, provider change, or query rewrite.

• Prompt sample: which customer questions did the tool choose, and how were they weighted?
• Query sample: which narrower searches or entities did the answer product derive?
• Candidate sample: which index, search provider, corpus, and fetched passages were available?
• Answer sample: which selected evidence was summarized, omitted, or cited?

3. Context changes what “the same query” means

A request such as “best accountant near me” is not the same experiment in Austin, Atlanta, and Anchorage. Language, country, city, device, account state, saved memory, conversation history, and explicit instructions can alter query interpretation and retrieval. OpenAI’s own ChatGPT Search guidance says it may use location inferred from an IP address and may use relevant saved memories when rewriting a query.

A vendor can choose a sensible default context, but the default is still a context. A US proxy is not “the internet.” An English prompt is not equivalent to a Spanish prompt translated after collection. A clean browser is not equivalent to a logged-in customer’s conversation. Scores from tools with different context controls are not directly interchangeable without a calibration study.

4. Models, indexes, and interfaces drift

The system under observation changes. Models are updated or routed differently. Search indexes add and remove documents. Retrieval providers change ranking and freshness behavior. A citation parser changes how it resolves redirects or source cards. An answer interface changes which citations it displays. A week-over-week movement can therefore reflect a brand change, a platform change, or a measurement change.

This is why a trend needs a control panel. If every tracked brand moves on the same day, suspect a product or collection change before celebrating a content win. If only one question moves after a source correction, the observation may be highly actionable—but it still should not be generalized to an invisible universal rank.

5. Citation visibility is not citation truth

A displayed source is evidence about an interface’s output, not a complete transcript of everything the system considered. OpenAI warns that search results and citations can be incomplete, outdated, or incorrect. Google’s grounding documentation exposes source metadata in supported API responses, but metadata still does not prove that every generated claim is supported. A tool that records only whether a URL appeared loses the distinction between direct support, partial support, and an unrelated citation placed near the claim.

Citation volatility adds another layer. A source can remain online while moving in and out of retrieved candidates. A page can be cited for one claim today and replaced by a more recent or more locally relevant page tomorrow. “Cited” is a useful observation; “trusted by the model” is usually an inference and should be labeled as one.

6. Vendor methodology is part of the result

Every ranking tool defines the experiment through choices that may not be visible in its headline score: prompt generation, question deduplication, weighting, model mode, location, retry policy, completion failures, mention matching, competitor definitions, and citation parsing. Those choices are not necessarily flaws. They become a problem when two scores look comparable while estimating different quantities.

A transparent vendor should make it possible to inspect enough of the method to reproduce or challenge a result. If the tool cannot export raw prompts, answer text, timestamps, product settings, and source URLs, the score may still be operationally useful, but it is weak as an auditable scientific measurement. The same standard applies to an in-house dashboard.

measurement_contract:
  question_set: "commercial-panel-v3"
  products: ["ChatGPT Search", "Google AI Overviews", "Perplexity"]
  market: "United States"
  language: "en-US"
  run_policy: "3 repeats per question; save every response"
  presence: "brand or owned product named in answer"
  citation: "visible URL supports the scored claim"
  failures: "record timeout or blocked run; never count as absence"
  report: "show n, date range, raw range, and methodology changes"

How to use these tools without fooling yourself

The practical answer is not to abandon third-party observability. It is to use it as a sensor with a known operating envelope. Pick the questions that matter to customers, keep their wording and version history, and separate branded, category, comparison, pricing, and problem-solving intents. Do not let an automatically generated prompt list become your definition of the market.

• Freeze a representative panel before looking at the next score. Include high-value questions where the correct answer could be nuanced or “it depends.”
• Record product, model or mode when exposed, account state, geography, language, timestamp, prompt, complete answer, and every visible citation.
• Repeat a control subset. Report the range or distribution, not only the latest run.
• Score presence, factual accuracy, recommendation, citation support, and competitor context separately.
• Keep collection failures explicit. A timeout, blocked request, or parser error is not zero visibility.
• Annotate deployments, source corrections, vendor changes, and prompt-set edits on the chart.
• Audit a sample manually. Automation can normalize URLs and flag matches; trained reviewers should confirm whether a source supports a claim.

A compact evaluation protocol

Before purchasing or trusting a tool, run a small parallel study. Give each candidate tool the same versioned panel. Export raw observations where possible. Compare not just the headline score, but question coverage, completed-run rate, repeatability, citation agreement, and the number of observations you can independently verify. Ask each vendor what changed when its score moved.

• Use at least 20–30 questions for a pilot, balanced across intent; expand before making market-level claims.
• Run repeated observations on a smaller control panel during the same collection window.
• Calculate agreement between the tool label and a human rubric for presence and citation support.
• Compare the raw answer and source URLs, not just screenshots or aggregate charts.
• Prefer a tool that exposes uncertainty and method changes over one that promises a precise universal rank.

This protocol does not make the answer ecosystem deterministic. It tells you how much confidence to place in a particular observation and whether a tool is fit for your decision. NIST’s AI measurement guidance makes the same general point: metrics need realistic, representative test sets and documented methodology, and their limitations of generalization should be recorded.

The right mental model: an observability system

Observability systems do not claim to be the system they observe. They provide signals about behavior under specified conditions. AEO and GEO tools are most valuable in this role: they can show that a key question is repeatedly answered with an outdated price, that competitors occupy a source pattern, or that a correction changes citation behavior after a release.

Treating the tool as an observability system also creates a better operating loop. Observe the failure, inspect the source and context, ship a technical or editorial fix, rerun the controlled panel, and check whether the outcome persists. The aim is not to win an imaginary universal leaderboard. It is to make important answers more accurate, attributable, and useful for real customers.

This article distinguishes documented product behavior from measurement inference. Vendor interfaces, models, retrieval systems, and indexes change; preserve raw observations and verify current documentation before treating a result as a specification.