How we measure LLM brand visibility

We curate a representative set of natural prompts that reflect how potential customers, journalists, analysts, and enthusiasts actually ask about a brand. Each query carries:

• Category: e.g. direct brand, performance, style, comparisons, skepticism.
• Priority weight: 1–5, reflecting business impact. Weight-5 queries count five times more in aggregate metrics.
• Query style: structured (research-style phrasing) or consumer_voice (natural language). LLMs often behave differently on the two, so we track both.

For Nike, we ship 10 categories spanning direct-brand awareness through comparison, innovation, sustainability, and skepticism-bait queries.

1. Each query is sent to every configured run target (provider/model pair) — or replayed from fixture in this demo. See section 3 for how multi-model output is aggregated.
2. URLs are extracted from the response and normalised to domains.
3. Domains are classified by rule into source_type (owned_brand, editorial, retailer, forum, wikipedia, review, social, government) and quality_tier.
4. A deterministic analysis engine runs regex-based checks for brand presence, mention rank, directness, competitor mentions, framing tags, and owned-source support.
5. A composite visibility score (0–100) is computed from the analysis output.
6. Typed recommendations are generated from the analysis using readable rules.

No LLM-as-judge. The only LLM call is the one that produced the response under analysis. Everything after that is readable TypeScript you can step through.

We run the same query against multiple LLMs (e.g. Claude, OpenAI) and store every run independently. Because each provider produces a different answer, mixing them in a single number would silently average behaviour you actually want to compare.

Headline KPIs use a single primary target. The primary target is the first entry of DEFAULT_RUN_TARGETS. Weighted Visibility Index, mention rate, first-mention rate, owned-source penetration, category performance, top opportunities, query-style breakdown, and movers all reflect this single target. The dashboard header surfaces which target is in use.

Multi-provider views span every target. The 30-day trend renders one line per target; the Model Performance panel compares every target head-to-head; the query detail page exposes a tab strip so analysts can compare providers run-by-run for one query.

A switcher could replace this rule with per-provider headline KPIs later. We keep the policy in one place — getPrimaryRunTarget() — so swapping it is a single-line change.

WVI is the priority-weighted average of the latest Query Visibility Score across all active queries:

WVI = Σ(score × priorityWeight) / Σ(priorityWeight)

A weight-5 query counts five times more than a weight-1 query. This ensures executive metrics reflect business priority, not query count.

Framing tags capture the sentiment and narrative lens around how the brand is discussed. They are the most nuanced signal: positive framings reward the score, mixed framings are neutral or slightly negative, and negative framings reduce it.

See the full dictionary with score impacts in the Settings page.

Three scopes of recommendation are generated:

• Query-level, generated immediately after each run from deterministic rules.
• Cluster: pattern detection across the latest run per query (e.g. repeated absence, repeated negative framing in a category).
• Global: threshold-triggered across the whole query universe (e.g. overall absence rate > 40%).

Each recommendation carries an evidence payload linking back to the specific runs or queries that triggered it, never a black box.