Wizard of Oz test (Wizard of Oz prototyping)
Users see a working service. Behind the curtain, a person is doing the work by hand.
- Time
- 60 min
- People
- 3–5
- Kind of work
- Test
In short
A Wizard of Oz test lets you learn whether a service works before you build the technology. Users interact with a simple interface, such as a chat window or form, while a hidden team member produces the replies or results by hand. You set one learning question, run a few sessions, record what happens and tell participants how it worked.
What it is
The name comes from the film in which a powerful wizard turns out to be a man behind a curtain. In a Wizard of Oz test, the user sees what looks like a working system: a chat assistant, a smart form, an automatic recommendation. In fact a team member, the 'wizard', reads each input and types or selects the response in real time.
This lets you test services that would be expensive to build, such as anything using artificial intelligence, complex rules or data from several systems. You learn what users actually ask, what answers they need and how they react, before investing months in development. Because the user does not know a person is responding during the session, you must handle consent and disclosure carefully. Tell participants in advance that they are testing an early version, avoid collecting real personal data, and explain the setup at the end.
When to use it
Use it when
- You are considering a chatbot, a smart assistant or automated advice and want to know what users will ask.
- The technology is expensive or uncertain, and you need evidence before committing budget.
- You want to test the content and tone of automated replies before writing rules or training a model.
- You need to find out which questions can be automated and which need a human.
Skip it when
- The test would involve real decisions about people's rights, money or health. Keep a Wizard of Oz test away from live cases.
- You cannot reply fast enough by hand to make the experience believable.
- Users would feel deceived in a way that harms trust, for example in a sensitive public service. Use a concierge test, where the manual service is visible.
You need
- Time
- About 60 minutes to set up and run 3 to 4 short sessions, plus a debrief
- People
- A facilitator with the user, one or two wizards out of sight, a note-taker, and 4 to 6 participants
- Materials
- A simple front-end (a chat tool, a shared form, a basic mock screen), a hidden laptop for the wizard, a prepared answer sheet, a consent form, recording if participants agree
How to run it
Step 1: Define the learning question
Write one question the test must answer, for example 'What do business owners ask when renewing a licence, and can a short answer resolve it?' Add a hypothesis: 'At least 7 in 10 questions can be answered from our existing guidance.'
Step 2: Build the visible service
Set up the thinnest possible interface: a chat window on a laptop, a form that sends to the wizard, or a printed screen updated by hand. It should look plausible but needs no real logic. Test that messages reach the wizard and replies come back.
Step 3: Prepare the wizard
Give the wizard a short guide: the tone to use, the sources to answer from, a target reply time (for example under 30 seconds), and what to say when they cannot answer. A prepared sheet of common answers keeps replies fast and consistent.
Step 4: Brief participants with care
Tell participants they are trying an early version of a service and that their feedback will shape it. Ask them to use invented details, not their real records. Get written consent for recording. Keep the full setup for the debrief, but never mislead people about anything that affects them.
Step 5: Run the sessions
Give each participant a realistic task. The wizard responds in real time while the note-taker records every question, reply, delay and reaction. Keep each session to 10 to 15 minutes. Watch for moments of confusion, repeated questions or people giving up.
Step 6: Disclose and debrief
After each session, explain that a person was producing the replies and why you tested it this way. Ask how they felt about the answers and whether knowing changes their view. Offer to delete their recording if they wish.
Step 7: Analyse and decide
Fill the worksheet: hypothesis, visible service, manual operation, observation. Group the questions users asked, note which ones the wizard answered easily and which needed judgment. Decide whether the hypothesis holds and what to build, change or drop.
Worked example
A chamber of commerce is considering a chat assistant to answer questions from small shop owners about renewing their trade licences. Before paying for the technology, a small team runs a Wizard of Oz test with six shop owners at a business centre.
The hypothesis: 'At least 70% of renewal questions can be answered from the existing guidance pages in under 30 seconds.' The visible service is a chat window on a tablet. Two staff members sit in the next room as wizards with the guidance pages open.
Participants are told they are testing an early version of a help service and are asked to use an invented shop name. Each is given a task: 'Your licence expires next month. Find out what you need to renew.'
Across six sessions, participants ask 41 questions. The wizards answer 26 directly from the guidance, about 63%. The remaining 15 need judgment, mostly about shops with several licences that expire on different dates. Two participants ask the same thing three times in different words, which shows the first reply was unclear.
In the debrief, most participants say the speed mattered more than knowing whether a person or system replied. The team concludes the hypothesis is close but not met. They decide to rewrite the guidance for multi-licence shops, then repeat the test before spending on the assistant.
IllustrativeTips and mistakes
Tips
- Use two wizards if replies are complex: one finds the answer, the other types it.
- Log the time of every reply. Delays reveal which questions are hard to automate.
- Keep a list of questions the wizard could not answer. It is often the most useful output.
- Rehearse with a colleague before the first real participant.
Common mistakes
- Testing with real personal data or live cases, which creates privacy and fairness risks.
- Skipping disclosure at the end, which damages trust once people find out.
- Letting the wizard improvise answers that the real system could never give.
- Running the test without a clear hypothesis, so you collect impressions instead of evidence.
What you walk away with
You walk away with a list of real user questions, evidence of which ones can be answered simply and which need human judgment, and a clear verdict on your hypothesis. You also know the tone and content users expect. If the service looks promising, write a formal hypothesis test for a larger pilot, or run a concierge test to deliver the full service manually to a few users.
Questions people ask
Is a Wizard of Oz test ethical?
It can be, if you handle it with care. Tell people in advance they are testing an early version, avoid real personal data or decisions, explain the setup afterwards, and let them withdraw their data. In sensitive services, prefer a concierge test where the manual work is visible.
What is the difference between a Wizard of Oz test and a concierge test?
In a Wizard of Oz test, users think they are using a system while a hidden person does the work. In a concierge test, users know a person is helping them directly. Wizard of Oz tests the interface and automation; concierge tests whether the service itself solves the need.
How many sessions do we need?
Four to six sessions usually show the main patterns in what users ask and expect. If you are estimating a percentage, such as how many questions can be automated, aim for at least 40 user questions in total.
Tool card
The worksheet your team fills in, one box per field.
- Hypothesis
- Visible service
- Manual operation
- Observation