Judging criteria that pick the right winner

Good hackathon judging criteria come straight from the purpose of the event: if you want something to pilot, weight feasibility and evidence; if you want new thinking, weight originality and insight. Use four or five criteria, weight them, write a short scoring guide for each, publish them with the challenge and calibrate your judges before the first pitch. The steps below show how.

Why judging goes wrong

Most judging problems are design problems, not people problems. The usual pattern: five criteria copied from another event (innovation, impact, feasibility, design, presentation), equal weights, a 1–10 scale nobody defined, and judges who meet for the first time ten minutes before the pitches. The result is that the most confident presenter wins, judges disagree for reasons they cannot articulate, and the team with the most promising idea leaves wondering what they did wrong.

There is also a well-documented bias at work. Psychologist Edward Thorndike described the halo effect in 1920: when raters form a strong impression of one quality, it colours their ratings of everything else. A polished pitch makes the idea look more feasible, the team more capable, the impact larger. Good criteria and a disciplined process are your main defence.

Seven steps to criteria that work

1. Start from what you will do with the winner. Go back to the purpose of the event and the owner's commitment (see how to write a challenge statement people want to solve). If the owner will pilot the winner in ninety days, the criteria must reward ideas that can actually be piloted in ninety days. If the goal is to surface new ways of seeing a problem, reward insight and originality and accept that the winner may be rough. Write the purpose at the top of the scoring sheet so judges see it every time.

2. Choose four or five criteria, no more. Judges cannot hold eight criteria in mind during a five-minute pitch. Typical building blocks:

• Problem understanding. Did the team find something true about the people affected?
• Value. If it works, how much does it change for those people or for the organisation?
• Feasibility. Can it be built and run within the stated constraints?
• Evidence. What did the team test, with whom, and what did they learn?
• Originality. Is it a genuinely different answer, or a version of what exists?

Notice what is missing: "presentation". Clarity matters, but as a criterion of its own it rewards performance over substance. If you include it, give it the smallest weight.

3. Weight them according to the purpose. Equal weights are a decision in disguise, usually the wrong one. A pilot-focused event might use: value 30%, feasibility 30%, evidence 25%, problem understanding 15%. An exploration event might use: problem understanding 35%, originality 35%, value 20%, feasibility 10%. A weighted decision matrix is the simplest way to lay this out.

4. Write what each score means. "Feasibility: 1–5" invites five different interpretations. Write a short anchor for the low, middle and high scores. For example, for evidence: 1 — no contact with users or data; 3 — talked to a few users or tested one assumption; 5 — tested the riskiest assumption with real users and changed the idea as a result. Anchors like these turn opinion into judgement and make disagreement useful, because judges can point to the description. An assumption map and a hypothesis test are what you are asking teams to do; say so in the anchor.

5. Publish everything with the challenge. Teams should see the criteria, weights and anchors on day one. This is not a courtesy; it is how you steer the work. If evidence counts for a quarter of the score, teams will go and find users. If you announce the criteria at the pitch, they will have optimised for slides. This is one of the 9 hackathon mistakes organisers keep repeating.

6. Calibrate the judges before the pitches. Brief judges the day before, or at least an hour before. Walk through the purpose, the criteria and the anchors. Then score one sample, a past submission or an invented one, separately, compare the scores and discuss the gaps. Ten minutes of calibration removes most of the disagreement you would otherwise have at the end. Ask judges to declare any conflicts of interest, and make sure at least one judge is the person who will own the winning idea.

7. Judge the work, not only the pitch. Give judges time with each team before the final presentations: ten minutes at their table, looking at the prototype and asking what they tested. Have judges score independently first, then deliberate. Discuss the teams where scores differ most, not the ones where everyone agrees. Use the total as the starting point for the decision, not a machine that makes it; if the panel overrides the numbers, write down why. Afterwards, give every team short written feedback against the criteria. A feedback capture grid keeps it structured.

A sample scoring sheet

Adapt the weights to your own purpose. Score the submission, never the individuals on the team.

Do's and don'ts

Do

• Derive every criterion from the purpose of the event.
• Keep the panel small: three to five judges.
• Score independently before discussing.
• Record the reasons for the final decision, especially when it departs from the totals.
• Give every team feedback, not only the winners.

Don't

• Don't let the most senior judge speak first in deliberation; others will anchor on their view.
• Don't add a "wow factor" or "judges' discretion" criterion; it reopens the door to the halo effect.
• Don't change criteria during the event.
• Don't use the impact-effort matrix as the final scoring tool; it is useful for a quick first sort of many entries, not for choosing one winner.

Illustrative example

Illustrative. A mid-sized insurer in Southeast Asia ran an internal hackathon on reducing the time it takes to settle small motor claims. The head of claims had agreed to pilot the winner in one region.

At the first event, the panel scored five equal criteria on a 1–10 scale with no definitions. A team with a striking animated pitch for a fully automated claims app won. When the head of claims looked closely, it required a core-system change outside the pilot budget. It went nowhere.

For the next event, the organisers rebuilt the sheet around the pilot: feasibility within the region's existing systems 30%, value to claimants 30%, evidence 25%, problem understanding 15%, with written anchors. They published it with the challenge and ran a twenty-minute calibration with the five judges using an invented entry. Judges spent eight minutes at each table before the pitches. The winning team had interviewed six claimants and three adjusters, discovered that most delays came from waiting for a single repair estimate, and proposed letting approved garages upload estimates directly. Their pitch was plain. It went into the regional pilot the following month.

Questions people ask

What are common hackathon judging criteria? Most events use some mix of problem understanding, value or impact, feasibility, evidence or testing, and originality. The right mix depends on what you will do with the winner, so choose and weight them from your purpose rather than copying another event's list.

Should presentation be a judging criterion? Rarely as a standalone criterion. If you include it, give it a small weight. Clarity of communication matters, but rewarding it heavily favours performers over teams with the strongest work.

How do you stop judges disagreeing? Write anchors for each score, calibrate on a sample entry before the event, score independently, then discuss the biggest gaps. Disagreement that remains after that is usually useful information about the ideas.

Where to go next

For the wider design of the event, see how to run a hackathon that produces something real and the full map in hackathons, jams and challenges: every type explained. The weighted decision matrix, assumption mapping and feedback capture grid guides give you the structure for scoring, for what teams should test and for feedback afterwards.