Testing an experience before you build it: role-play, Wizard of Oz and concierge tests

You can test a customer experience long before any system, staff training or fit-out exists. Act it out with your team to find the awkward moments (role-play), fake the technology with a person behind the curtain while real customers use it (Wizard of Oz), or deliver the whole service by hand to a few real customers (concierge). Each answers a different question, so choose by what you most need to learn, and decide what would count as a pass before you start.

This piece is about experiences: services, journeys and moments where people, timing and feeling matter as much as features. For testing ideas in general, start with How to test an idea before you build it.

Why experiences need their own kind of test

A product prototype shows you whether people can use a thing. An experience prototype has to show you something harder: whether the sequence of moments works when real people are tired, rushed, anxious or accompanied by family. It also has to test the staff side: who does what, when the handover happens, what information the next person needs.

You cannot see these on a screen mock-up. A welcome that reads well on paper can feel cold said aloud; a three-step process can collapse because step two depends on a colleague in another building. Experience tests surface such failures while they are cheap to fix.

Choose the right test

The Wizard of Oz name is usually credited to the researcher Jeff Kelley, who used the technique in the early 1980s to study how people would talk to computer systems that did not yet exist. The concierge test was popularised in lean startup practice as a way to serve the first customers by hand before automating anything.

A useful rule: role-play tests the script, Wizard of Oz tests the front, concierge tests the offer.

How to run an experience test: seven steps

1. Pick the moment that worries you most

Do not test the whole journey. From your journey map or service blueprint, choose the one or two moments where the experience is most likely to fail or matter most: the first contact, a handover between teams, the moment bad news is delivered, the wait. Test those.

2. Write what must be true and what would count as a pass

Write the belief behind the moment as a hypothesis: "If a nurse calls patients the day after discharge, most will raise a concern they would otherwise have taken to the emergency department." Then set the pass line before anyone takes part. The hypothesis test guide has a template. Without a pass line, every result looks encouraging.

3. Script both stages

Experiences have a front stage, what the customer sees, and a back stage, what staff and systems do to make it happen. Write both. A simple storyboard of six to eight frames is enough for the front stage. For the back stage, list each step, who does it and what information they need. In a Wizard of Oz test, the back-stage script is what your hidden operator follows, so make it precise: response times, wording, what to do when a request is outside the script.

4. Build only what the moment needs

Role-play needs a room arranged roughly like the real space and a few props. Wizard of Oz needs a believable front, such as a chat window, a phone number or a kiosk screen, and an operator who can respond quickly. A concierge test needs a simple way to take requests and a person with time set aside. Fidelity should match the question, not your pride.

5. Rehearse once, then run with the right people

Rehearse once with colleagues to catch gaps in the script. Then run it for real:

• Role-play with your own team first. Swap roles so the person who designed the service plays the customer. Then, if possible, run it with two or three real customers acting as themselves. The role-play guide covers how to brief players.
• Wizard of Oz with five to ten real customers using the front as they would the finished service. The Wizard of Oz test guide covers set-up and operator rules.
• Concierge with a handful of real customers over one to four weeks, long enough to see repeat use. The concierge test guide has a tracking sheet.

6. Watch for the signals that matter

Assign one person only to watch. In experience tests the most important signals are often physical or emotional, so note:

• Hesitations, repeated questions and the moment people ask "what happens now?"
• Where staff, real or acting, have to improvise because the script ran out.
• Whether people come back, recommend it or ask to keep using it (concierge).
• How long the back stage really took compared with your plan.

Capture reactions straight after each session with a feedback capture grid: what worked, what didn't, questions raised, new ideas.

7. Decide and update the blueprint

Compare results with your pass line and decide: build, change and test again, or stop. Either way, update the service blueprint with what you learned, especially the back stage. A concierge test almost always reveals steps, delays and dependencies that nobody had drawn. That is the most valuable output: an honest picture of what the service takes to deliver. For choosing between a journey map and a blueprint to hold this, see Journey map or service blueprint? When to use which.

Illustrative example

Illustrative: an invented organisation, for explanation only.

A regional hospital in a mid-sized European city wanted to reduce readmissions among older patients after discharge. The proposal was a digital "recovery companion": an app that would check in daily and route concerns to the right clinician. Building it would take a year.

In a morning of role-play first, nurses played patients and patients' adult children played themselves. It showed at once that the discharge conversation was too rushed for anyone to remember instructions, and that the person who would answer the check-ins was usually a relative, not the patient.

Then a three-week concierge test: two nurses phoned twenty discharged patients each morning, following a short script. The pass line, set in advance, was that at least a third of calls would surface a concern the team could resolve without a hospital visit. The nurses logged every call and how long it took.

The result met the pass line, but the log showed something more useful: most concerns were about medication timing, and nearly all could be answered by a pharmacist rather than a nurse. The hospital changed the design before building anything: a pharmacist-led call in week one, then a simpler digital check-in. The app brief became much smaller.

Do's and don'ts

Do

• Test the riskiest moment, not the whole journey.
• Write the pass line before the first session.
• Script the back stage as carefully as the front.
• Let the designer play the customer at least once.

Don't

• Explain the experience to participants while they are in it. If you have to explain, that is a finding.
• Keep a Wizard of Oz test secret afterwards. Tell participants a person was involved once the session ends, and never collect sensitive personal data through a faked system without proper consent and safeguards.
• Run a concierge test with friends of the team; they will be kind.
• Treat a good role-play as proof of demand. It tests the script, not whether people want the service.

Questions people ask

How do you prototype a service? Act out the key moments with simple props (role-play), fake the hard-to-build parts with a hidden person while real customers use it (Wizard of Oz), or deliver it by hand to a few customers (concierge). Choose by what you need to learn.

Is a Wizard of Oz test ethical? It can be, if participants are not put at risk, you tell them afterwards that a person was involved, and you handle any data as carefully as a real service would. Avoid it for health, financial or legal decisions where a mistake would cause harm.

What is the difference between a concierge test and a pilot? A concierge test is small, manual and meant to learn whether the service is worth building and what it takes. A pilot runs a near-finished version at limited scale to check it works before full rollout.

Where to go next

Choose your test with the role-play, Wizard of Oz test and concierge test guides, and hold what you learn in a service blueprint. For where experience testing fits in building a customer-centric organisation, read Customer-centric is a practice, not a slogan: where to start. For the research that tells you which moments to test, see How to really understand your customers.