How we measure

This page describes in short how an assessment is made. Details are documented in the master's thesis.

Which AI service

We measure ChatGPT only. Questions are sent through the official OpenAI API (Responses API with web search). This is an API-based approximation of ChatGPT, not a replica of the ChatGPT app. Answers in the app can differ. Google Gemini and Perplexity are not measured.

Which questions

Each assessment uses 20 questions. 10 contain the name of the business, for example which services it offers or where it operates. 10 do not contain the name. They ask, as a potential customer would, for providers in a category and location. Questions are asked in German by default. A person reviews the questions before they are sent to the AI service. The question set is then frozen and not changed. If the same business is measured again, the same question set is used.

How answers are evaluated

Statements about the business are compared with a set of verified facts from the questionnaire and from the business's own website. Every figure is shown with its numerator and denominator, not only as a percentage:

Factual accuracy: correct statements divided by all checked statements (correct and incorrect). Statements that cannot be verified are shown separately.
Coverage: expected facts stated correctly divided by all expected facts.
Visibility: answers to the questions without the business name that name the business, divided by all of these answers that name at least one provider or platform.
Other providers named and sources cited

Figures with a denominator below 5 are marked as limited data.

AI use and human review

Statements are extracted and checked with an AI model (Anthropic Claude). A person reviews the questions before the measurement and every report before it is released. Reports are labelled as AI-generated.

Limitations

AI answers change over time. The search results behind the services cannot be fixed.
The location is stated only in the question text.
Results of a free assessment are preliminary. The reference facts come from the questionnaire and the business's own website. A person reviews them, but they are not independently verified.
An assessment makes no statement about future visibility, revenue or customer numbers.
Only ChatGPT is measured. Google Gemini and Perplexity are not used. The results make no statement about Google AI answers or other AI services.