LAB 14 · CAN AN AI JURY JUDGE HOW A ROOM LOOKS?

Can an AI jury judge how a room looks?

A jury of AI models scored pictures of the rooms of a 3D presentation space, blind, on one rubric, twice. Every juror that could score our calibration pair told a good picture from a deliberately ruined one. On eleven rooms that did not change between the two sittings, the jury's median was 3.857 out of 10 the first time and 4.0 the second, with four jurors and then three. Single rooms moved by up to 0.572.

So a difference between two rooms smaller than that is not, on these two rounds, a verdict on the rooms. This page shows numbers only: no picture.

What it is.

The lab in plain words, and how it runs.

Can a jury of AI models judge how a room looks, and does it give the same answer twice? The rooms are eleven rooms of a 3D presentation space, and the jurors are AI models, not people: no person scored a picture.

Each juror scored pictures of the rooms one at a time, blind, on one rubric of seven criteria. To count, a juror first had to tell a good picture from a deliberately ruined copy of it, a pair we made ourselves, by at least 1.5 points; a juror that could not was off the jury for that round. We did this in two rounds, on rooms that were not reworked in between, though each was photographed again.

The median of the rooms was 3.857 out of 10 the first time and 4.0 the second, with four jurors and then three; single rooms moved by up to 0.572. Only these two rounds are shown. Two more rounds, 2b and 2c, are left out: they ran on a lane only, never on the private main.

How it works: 11 rooms, scored blind by an AI jury in 2 rounds
  1. 11 rooms, the same in every roundeach juror saw one picture at a time with the rubric alone: no build, version, round or room name
  2. The calibration pair, made by usa known-good reference picture we rendered, and the same picture deliberately ruined
  3. One rubric, 7 criteriaeach scored 1 to 10

Round 1: 4 AI models tried for the jury

  1. The calibration gate4 of 4 passed
  2. 4 jurors score the 11 roomsAI models, not people; each room gets the median of its jurors' scores
  3. The median of the rooms: 3.857 out of 10the jury ranked the pair by 3.5, so the round counts

Round 2: 4 AI models tried for the jury

  1. The calibration gate3 of 4 passed; Qwen3.8 Flash (Qwen) gave no usable score for the good picture and was taken off
  2. 3 jurors score the 11 roomsAI models, not people; each room gets the median of its jurors' scores
  3. The median of the rooms: 4.0 out of 10the jury ranked the pair by 4.0, so the round counts

The jurors.

AI models, not people.

A SEPARATE AI REVIEWER, NOT A HUMAN

Every score on this page is an AI model's. No person scored a picture.

  • Claude Haiku 5.5 (Anthropic).
  • GPT-6 Luna (OpenAI).
  • Gemini 3.8 Flash (Google).
  • Qwen3.8 Flash (Qwen).
  • Gemma 3 4B, run locally: a small local model, scored as a floor for comparison and never on the jury.

Each juror saw one picture at a time with the rubric alone: no build, version, round or room name; the pictures were re-encoded under random names, in a different order for each juror, with the calibration pair mixed in.

How the jury works.

  • The rubric. Seven criteria, each scored 1 to 10: composition; typography; lighting; materials and texture; style consistency; readability on a projector from 10 m; whether a board would take it seriously as a product presentation.
  • The score. A juror's overall score for a picture is the mean of its seven criterion scores; a picture's score is the median of the jurors' overall scores.
  • The calibration gate. Every juror also scores the calibration pair. A juror whose good picture does not score at least 1.5 above the bad one is out of that round's jury. A round with fewer than three jurors left, or whose jury does not rank the pair by that margin, is void.
  • The rooms. Eleven rooms of a 3D presentation space, each staging one slide of a talk; none was reworked between the rounds, but each was photographed again for round two.

The calibration gate.

Each round, each juror, on the calibration pair.

AN AUTHORED FIXTURE, WRITTEN BY US

The calibration pair is ours: a known-good reference picture we rendered, and the same picture deliberately ruined. It is the yardstick, not a room.

RoundModelGood pictureRuined pictureMarginJury
1Claude Haiku 5.5 (Anthropic)6.1431.4294.714on the jury
1GPT-6 Luna (OpenAI)7.02.8574.143on the jury
1Gemini 3.8 Flash (Google)5.1433.1432.0on the jury
1Qwen3.8 Flash (Qwen)4.7141.1433.571on the jury
1Gemma 3 4B, run locally6.7143.03.714floor, never on the jury
2Claude Haiku 5.5 (Anthropic)6.2861.5714.715on the jury
2GPT-6 Luna (OpenAI)6.5712.2864.285on the jury
2Gemini 3.8 Flash (Google)5.1433.02.143on the jury
2Qwen3.8 Flash (Qwen)no score1.0noneout
2Gemma 3 4B, run locally7.02.7144.286floor, never on the jury

In round 2, Qwen3.8 Flash (Qwen) was out. It gave no usable score for the good picture: its two replies on it could not be read, so it had no margin, and the gate took it off the jury. Round 2 was judged by the three that passed.

Each juror's overall score for each picture of the pair. Every round passed the gate: round 1's jury ranked the pair by 3.5, round 2's jury ranked the pair by 4.0.

The rooms, round by round.

Each room's score in each round, from the record.

The same eleven rooms in both rounds, the jury's median score for each, out of 10. Eleven rooms of a 3D presentation space, each staging one slide of a talk; none was reworked between the rounds, but each was photographed again for round two. A room scored by fewer jurors than the round's jury is one whose other replies could not be read (the record names whose).

RoomRound 1, Round 2, Change
Room 13.785 (4 jurors)4.286 (3 jurors)+0.501
Room 23.714 (4 jurors)4.286 (3 jurors)+0.572
Room 34.357 (4 jurors)4.0 (3 jurors)−0.357
Room 43.857 (4 jurors)3.571 (3 jurors)−0.286
Room 53.857 (3 jurors)3.857 (3 jurors)0.0
Room 64.072 (4 jurors)3.857 (3 jurors)−0.215
Room 73.929 (4 jurors)4.0 (3 jurors)+0.071
Room 84.215 (4 jurors)4.571 (3 jurors)+0.356
Room 94.0 (3 jurors)4.0 (3 jurors)0.0
Room 103.785 (4 jurors)4.143 (3 jurors)+0.358
Room 113.643 (4 jurors)3.571 (3 jurors)−0.072
The median of the rooms3.8574.0+0.143

Recounted at build from each juror's score: the build refuses the record if a room's median is not its jurors' median, or the round's is not its rooms'.

What it says.

  • The gate works as a gate. It took a juror off when that juror could not score the yardstick, and the round went on with the three that could. Every juror that scored both pictures of the pair ranked the good one higher, by at least 2.0.
  • The jury is not a fine ruler. Between the two sittings five rooms went up and four went down, by up to 0.572, although none was reworked. The median moved from 3.857 to 4.0. Some of that is the jury changing (four jurors, then three) and some the pictures being taken again; two rounds cannot separate the two.
  • What follows. A difference smaller than that, between two rooms or before and after a rework, is not shown by this jury on two rounds. A bigger one may be.

What is left out.

  • Every picture. Each committed picture is a room of a talk themed for a client, so none is shown; the rooms are described in neutral words.
  • The jurors' reasons, the rubric's wording and the rooms' names: only numbers are vendored.
  • Ten other rooms that the same sittings judged before and after a rework: this lab is about the jury, so it shows only the rooms that stayed the same.
  • What the jury cost, by the site's rule on money.
  • Round 2b: run on a lane only, never on the private main. Not shown, not counted.
  • Round 2c: run on a lane only, never on the private main. Not shown, not counted.

Check it yourself.

Everything on this page comes from one record, published here and pinned by its sha256. It was read from the agent space's private repository: the design jury's two committed rounds, their scores and calibration, the rubric's rule and the aggregation code, read at the main head.

The commits it stands on

  • Last changed the rubric and the aggregation, . Private commit f2b53edf, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Last changed the calibration pair, . Private commit 9a79a6bc, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Committed both rounds' scores, . Private commit 60dfae05, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Committed the report and the comparison table, . Private commit 201fd650, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.

The files, by sha256

  • Round one's scores: every juror, every picture, the calibration: bfed03043c9034f5742ae3887e7c7e74eeb3924fa30c4fcf1ab153f076a4b5d6
  • Round two's scores, the same: 3492354519678b59332d97ad79f9226743eab46cfefe2dd066b8108f3128635d
  • Round one's own report, which says what each juror saw: 075c1b14cc31e96fe079d4899c913b30420ca42310e49fc0e6d9560b104f2f94
  • Round two's own report, the same: 6bd3e5cb4e4ff44b2389c9408661fc625ae1caabc758aa560b4dcf1d676edfc4
  • The comparison table of both rounds: 8de3a4a9611a9a1847f420f8620210ee43dfbe46d2e8d4debeb15af71d0fe93c
  • The jury's report of both rounds: 911b3d54fc5d293c04f45f6ebddd67527328afd604ff46ddd392e7e100c2e969
  • The jury's one rubric, with the calibration rule: c7d627f479dcb5dab2690bba454aa2a6d47e2664a4e7d52b4dfc66a4c248619d
  • The calibration pair, pinned by sha256: 8d5c3a7e38525ea0fad2ff14e9cb4cf59c7cf585b103b7323988bb1fd2281778
  • The code that turns scores into medians and the calibration verdict: e4754dd4ddf0ce559dd7e95c62c9e0f9bbac8a5505112f5fcab0a7bc47aa2c6f

Main was checked by git merge-base --is-ancestor of each cited commit against the private main head, in a clone of the private repository, on .

What this does not show.

  • Two rounds, on one day, of eleven rooms of one style. It does not say how this jury does on other pictures, or over more rounds.
  • The jurors are AI models; no person's judgement of these rooms is shown or compared.
  • It shows the noise between two sittings, not its cause: the jury and the pictures both changed.