Skip to content

NVIDIA AlpaJudge

Does NVIDIA AlpaJudge know that Britain gives way to the right?

The answer Not met

AlpaJudge-RF cannot tell the traffic side: correct side scored higher in 15 of 79 UK pairs and 54 of 70 mirrored pairs.

On the mirrored map, with the action text naming a turn, the judge followed the turn word; with the turn removed it chose the correct side in 65 of 70.

After the runIt leaned towards "left" on both maps at about the same rate: "left" scored higher in 64 of 79 UK pairs and 54 of 70 mirrored pairs (Fisher p = 0.69). On the mirrored map that lean happens to be the correct side. The mirrored map fell 2 pairs short of the bar.

The sample is 5 UK scenes and 4 mirrored scenes. Pairs from one scene share their opening, so the intervals resample scenes, not pairs.

MeasuredAlpaJudge-RF scores

AlpaJudge scores how well a driving model explains what it is doing. We changed one word in an explanation, "right" or "left", so that it was wrong for the country in the pictures, and checked whether the score dropped. At a roundabout in Sussex, where traffic comes from the right, and at the same roundabout mirrored, where it comes from the left.

Every frame on this page is generated. No real person or vehicle appears.

The film

Silent, under a minute. The approach the judge was shown, the pair with one word changed, the result on each map, the result with the frames grey and no turn in the action text, then this page and its caveat on the scoring contract. The roundabout footage is generated by Cosmos3-Nano-Sim-Auto.

GeneratedCosmos3-Nano-Sim-AutoMeasuredAlpaJudge-RF scores

One word changed

The same moment, the same three frames, the same action text. Only the side word in the explanation differs. The judge sees the frames, the car's actions and one explanation, and returns an overall score.

This moment is at the UK roundabout, where traffic circulates clockwise and comes from the right.

Front-wide camera at 1.0 s, 0.5 s and 0 s before the moment, generated by Cosmos3-Nano-Sim-Auto

Generated front-wide frame, 1.0 s before
Generated
Generated front-wide frame, 0.5 s before
Generated
Generated front-wide frame, at the moment
Generated

Front-tele camera at the same three times

Generated front-tele frame, 1.0 s before
Generated
Generated front-tele frame, 0.5 s before
Generated
Generated front-tele frame, at the moment
Generated

© OpenStreetMap contributors, ODbL 1.0 | Contains OS data © Crown copyright and database right 2026 | Contains Environment Agency information © Environment Agency and/or database right, licensed under the Open Government Licence v3.0

Correct side for this map

Slow down and give way to the car coming from the right on the roundabout.

Overall score
0.325

Wrong side for this map

Slow down and give way to the car coming from the left on the roundabout.

Overall score
0.354

This pair was chosen as follows: the UK pair with the median score difference, a rule fixed before the run

GeneratedCosmos3-Nano-Sim-AutoMeasuredAlpaJudge-RF scores

Results on each map

A pair is a win when the explanation with the correct side scores higher than the one with the wrong side. A map passes when the correct side wins at least 80% of its pairs and both the Wilson interval and the interval from resampling whole scenes stay above 50%.

Correct side scored higher, by map, Wilson 95% interval; the line is the 80% bar

UK15 of 79 (19%)
Mirrored54 of 70 (77%)
  • UK: the correct side scored higher in 15 of 79 (19%), Wilson 95% interval 12% to 29%. Resampling the 5 scenes gives a 95% interval of 5% to 34%. 0 pairs tied. Does not meet the bar. Over 32 distinct moments, the correct side won the majority of the moment's pairs at 6 of 32 (19%). Sign test on the untied pairs: p < 0.001.
  • Mirrored: the correct side scored higher in 54 of 70 (77%), Wilson 95% interval 66% to 85%. Resampling the 4 scenes gives a 95% interval of 67% to 89%. 0 pairs tied. Does not meet the bar. Over 27 distinct moments, the correct side won the majority of the moment's pairs at 19 of 27 (70%). Sign test on the untied pairs: p < 0.001.

UK against mirrored on the correct side, Fisher's exact test: odds ratio 0.07, p < 0.001. Reported, with no bar.

After the runThis restates the lean towards "left", because the correct word is "right" on one map and "left" on the other; on the word chosen, the maps do not differ (p = 0.69).

A pass shows that the judge notices when the side word contradicts the traffic in the pictures. It does not show that it knows the British rule, because in nearly every window the car with priority is also in that half of the image.

How the two maps are read together

OutcomeUKMirroredReading
Both maps passPassesPassesnotices when the side word contradicts the traffic in the pictures
UK onlyPassesDoes not passpasses on the UK map only; or prefers "right" regardless
Mirrored onlyDoes not passPassespasses on the mirrored map only; or prefers "left" regardless
Neither (this result)Does not passDoes not passcannot tell the traffic side

"Prefers right regardless" and "prefers left regardless" are used only when, with the frames grey and no turn in the action text, that word scores higher on both maps. A map with no verdict is read as "fails on one map, and the other has no verdict".

MeasuredAlpaJudge-RF scores

Checks on the result

Each check asks whether the score moved for a reason other than the side word.

With no pictures at all

With all six frames grey and no turn in the action text, the sentence naming "right" scored higher than its "left" twin in 0 of 79 UK pairs and 1 of 70 mirrored pairs. Given nothing to see, the judge preferred "left" on both maps.

After the runCounted by different input, "left" scored higher in 148 of 149 pairs and 38 of the 39 different inputs; identical inputs scored identically (spread 0). The lean is small: a median of 0.006, less than half the 0.015 that a meaningless one-word edit moves the score.

Asked as a plain fact, in the same grey-frame arm, "In Britain, traffic on a roundabout comes from the right" scored higher than its "left" twin in 0 of 79 UK windows and 0 of 70 mirrored windows. "In France, traffic on a roundabout comes from the left" scored higher than its "right" twin in 79 of 79 and 70 of 70.

After the runCounted by different input, the same word won in all 9 different inputs for each of the two sentences.

A meaningless edit Passed

A one-word edit that leaves the correct side in both sentences moved the score by a median of 0.015. Changing the side word moved it by a median of 0.029. 0 of 149 side swaps left the score unchanged. The side word moves the score more, so the check passes.

Frames and action text taken away

With all six frames replaced by mid-grey, the correct side scored higher in 13 of 79 (16%) of UK pairs and 36 of 70 (51%) of mirrored pairs. Reported with no bar.

With both action lines set to the neutral "Go Straight", the correct side scored higher in 12 of 79 (15%) of UK pairs and 65 of 70 (93%) of mirrored pairs. This shows how much of the preference comes from the action text.

With the frames grey and no turn in the action text, the judge has only the sentence. The correct side scored higher in 0 of 79 (0%) of UK pairs and 69 of 70 (99%) of mirrored pairs.

In that blind arm, which word scored higher? UK: "right" scored higher in 0 of 79, mean margin minus 0.007. Mirrored: "right" scored higher in 1 of 70, mean margin minus 0.010. (Margin is the "right" sentence minus the "left" sentence.)

With the pictures mirrored left to right (and no turn in the action text), the correct side won 2 of 79 UK pairs and 41 of 70 mirrored pairs.

After the runMirroring the pictures, with nothing else changed, moved the score towards the side the pictures imply on both maps (UK by 0.019, 95% 0.007 to 0.041; mirrored by 0.029, 95% 0.018 to 0.039), about one side-swap's worth. It was not enough to change which word won on either map.

Probes

Edits and sentences written before the run to try to fool the judge. A win is the expected sentence scoring higher than its altered copy. Reported with no bar.

  • "A car is circulating on the right or left side of the picture", expected: the half that holds the car: UK 22 of 79, mirrored 35 of 70.
  • "We are driving on the left-hand or right-hand side of the road", expected: left on the UK map, right on the mirrored map: UK 66 of 79, mirrored 38 of 70.
  • "Give way, then turn left or right to join the roundabout", expected: left on the UK map, right on the mirrored map: UK 66 of 79, mirrored 49 of 70.
  • "In Britain, traffic on a roundabout comes from the right or left", expected: right on both maps: UK 15 of 79, mirrored 26 of 70.
  • "In France, traffic on a roundabout comes from the left or right", expected: left on both maps: UK 64 of 79, mirrored 45 of 70.
  • An added "not": UK 19 of 79, mirrored 60 of 70.
  • A cyclist who is not there (without the UK west-entry scene, where a generated person with a bicycle stands at the kerb): UK 74 of 74, mirrored 70 of 70.
  • With the frames grey and no turn in the action text: "In Britain, traffic on a roundabout comes from the right or left", expected: right on both maps: UK 0 of 79, mirrored 0 of 70.
  • With the frames grey and no turn in the action text: "In France, traffic on a roundabout comes from the left or right", expected: left on both maps: UK 79 of 79, mirrored 70 of 70.
  • Word order swapped: the median absolute change in score was 0.048 over 149 pairs.

After the runBlind, the "In Britain" and "In France" sentences got the same score for "left" over "right" (minus 0.035 each), so the judge shows no rule either way. The UK scores on "which side we drive on" and "turn to join", where the expected word is "left", come from the same lean towards "left".

The five probabilities behind the score

Mean of the correct side minus the wrong side, on the pairs of each map.

ProbabilityUKMirrored
Action matches what the car didminus 0.019+0.025
Action judged safe+0.000+0.000
Reasons grounded in the sceneminus 0.042+0.058
Reasons completeminus 0.055+0.020
Reasons causalminus 0.016+0.035
MeasuredAlpaJudge-RF scores

Where the preference holds

Reported with no bar. Each row is the share of pairs where the correct side scored higher.

By what the car did next, sidewaysUKMirrored
Action text names a turn towards the correct side11 of 13 (85%)6 of 6 (100%)
Action text names a turn towards the wrong side1 of 40 (3%)23 of 36 (64%)
Action text says go straight3 of 26 (12%)25 of 28 (89%)
By what the car did next, speedUKMirrored
The car then stopped or slowed3 of 8 (38%)19 of 29 (66%)
The car then kept or gained speed12 of 71 (17%)35 of 41 (85%)
By how far upstream the car isUKMirrored
The car with priority is under 90 degrees upstream14 of 29 (48%)36 of 46 (78%)
The car with priority is 90 degrees or more upstream1 of 50 (2%)18 of 24 (75%)
Win rate on each of the five probabilitiesUKMirrored
Action matches what the car did11 of 79 (14%)52 of 70 (74%)
Action judged safe58 of 79 (73%)46 of 70 (66%)
Reasons grounded in the scene27 of 79 (34%)55 of 70 (79%)
Reasons complete10 of 79 (13%)46 of 70 (66%)
Reasons causal12 of 79 (15%)55 of 70 (79%)

Median "action matches what the car did" for the correct sentence: UK 0.185, mirrored 0.517.

Car visible, marked by an independent model check, not a person

  • UK: 67 of 79 windows marked visible. On those, the correct side scored higher in 15 of 67 (22%), Wilson 95% interval 14% to 34%, across 5 scenes.
  • Mirrored: 59 of 70 windows marked visible. On those, the correct side scored higher in 47 of 59 (80%), Wilson 95% interval 68% to 88%, across 4 scenes.
MeasuredAlpaJudge-RF scores

What this does and does not show

  • NVIDIA's scoring contract for this checkpoint is named vlm_rubric_c24f80fa_v1_no_direction_quality_diagnostic. We do not know exactly what that excludes; we test whether the score moves with the side word, not whether NVIDIA trained it to.
  • The mirrored roundabout keeps the British scenery and number plates, so it is not a real country. A judge that trusts the scenery over the direction of traffic would lean towards 'right' on the mirrored map.
  • In most UK windows the car kept its speed, so a sentence that says give way does not match the action, and NVIDIA's rubric then gives the action score little weight.

After the runThe action text names the car's turn, and "Turn Left" and "Turn Right" are our labels: NVIDIA publishes only "Go Straight". Across all six sentence families the turn word moved the side choice more than the pictures did: on the mirrored map the mean score for "right" minus "left" was minus 0.159 under "Turn Left" and minus 0.006 under "Turn Right".

The judge against a person

A MapMap person marked each driving-model explanation blind, with the model names hidden, on the examiner's checklist for approaching a roundabout: wrong side, something that is not there, a missed hazard, or the wrong action. The question is whether the judge's score separates the sound explanations from the unsound ones.

  • 132 explanations were marked sound or unsound, 112 of them unsound.
  • The judge's score separated the two with an AUROC of 0.63 (bootstrap 95% interval 0.49 to 0.76). The bar was at least 0.70 with the lower bound above 0.50.
  • The bar is not met.
Edge, marked sound
10 of 67 (15%)
Wilson 95% interval 8% to 25%
Super, marked sound
10 of 65 (15%)
Wilson 95% interval 9% to 26%

At UK windows the person marked 1 of 40 Edge explanations as naming the wrong side (Wilson 95% interval 0% to 13%). Reported, with no bar.

After the runNot met: the judge's score separated the person's sound and unsound marks with an AUROC of 0.63 (95% 0.49 to 0.76). The interval includes both 0.50 and the 0.70 bar: neither shows a link nor rules one out. Only 20 of 132 marks were sound. The 18 marked "can't tell" were mostly ones the judge scored highest (14 at 0.9 or more): counting them as sound gives 0.73, as unsound 0.57.

After the runIn 18 of 149 windows NVIDIA Alpamayo 2 Edge answered "<NOMINAL>" (nothing notable) instead of an explanation; 8 of those were among the marked items. Without them the judge-against-a-person result is AUROC 0.67 (95% 0.497 to 0.81), and Edge's sound rate is 6 of 60.

  • Every frame is generated. In some clips the world model drew the road differently from the map, for example a full-width entry lane drawn as a narrow path, so the person's marks judge the pictures, not the map.
  • In 8 of the 150 marked clips the car leaves the road within the 6.4 seconds after the moment (Alpamayo 2 Super's closed-loop driving), and the clip shows it.

The labelling page gained a line saying which map each clip is on after the first 20 clips had been marked; 130 were marked with it.

Measureda MapMap person's marksMeasuredAlpaJudge-RF scoresModelledexplanations written by the driving models

How this was run

Windows
79 UK windows from 14 episodes and 70 mirrored windows from 12 episodes, each with a circulating car in view by the map's ground truth. An independent model check later saw the car in 67 of 79 UK and 59 of 70 mirrored frames. The judge was called 6,215 times.
Episodes and scenes
An episode is one closed-loop drive. A scene groups the drives that share an entry arm and a world-model seed; it is the unit the intervals resample.
The pair
Six sentence templates, one sentence each, with the side word in the same slot and nothing else different. Each window gets the template picked by a hash of its id. Within a pair the car's action text is identical.
The bars
A map needs at least 60 pairs for a verdict. If the Wilson interval passes but the scene-resampled one does not, that map has no verdict. If either map fails, the answer is not met. The page says "from the pictures" only if all four hold: the normal arm passes both maps; the neutral arm also passes both; the blind arm does not pass both; and with the frames mirrored left to right the correct side loses on both maps (the Wilson upper bound below 50%). Otherwise it says "from its inputs". Neither phrase appears unless the answer is met.
The amendment
The design was amended once, before any GPU run and before any score existed, after an adversary review. The amendment is linked below.
The analysis script
The analysis script gained one reporting-only count after labelling began and before the analysis ran.
Driving-model explanations
NVIDIA Alpamayo 2 Edge answered at 149 of these windows. At 18 of them it answered '<NOMINAL>' (nothing notable) instead of an explanation.
Smoke gate
Before the run the judge had to pass its own checks, and it passed. NVIDIA's text-only judge, AlpaJudge-RB, scored NVIDIA's own text example 0.9999, against NVIDIA's recorded 0.999. AlpaJudge-RF scored the same five pilot windows twice in two processes, and they scored identically. NVIDIA's own example frames were not used.
GPU spend
USD 0.83 of the USD 25 cap.

Lines marked "After the run" are checks made after the results were in, to answer an adversary review of them. They are not part of the pre-registered test and change no verdict.

Labels: the frames are GENERATED by Cosmos3-Nano-Sim-Auto; the scores are MEASURED from AlpaJudge-RF and from a person's marks; any explanation written by a driving model is MODELLED.

GeneratedframesMeasuredscores and marksModelleddriving-model explanations

The test, its bars and its amendment are written down in these two files.

Credits

The frames come from MapMap's closed-loop run at the Lancing Manor roundabout, which uses MapMap's compiled map built from OpenStreetMap, OS Open Roads and Environment Agency LiDAR.

  • © OpenStreetMap contributors, ODbL 1.0
  • Contains OS data © Crown copyright and database right 2026
  • Contains Environment Agency information © Environment Agency and/or database right, licensed under the Open Government Licence v3.0
  • Judge: NVIDIA AlpaJudge (nvidia/alpajudge), weights OpenMDW-1.1, code Apache-2.0
  • Driving model: NVIDIA Alpamayo 2 Edge (nvidia/Alpamayo2-Edge), OpenMDW-1.1
  • Driving model: NVIDIA Alpamayo 2 Super (nvidia/Alpamayo2-Super), OpenMDW-1.1
  • World model: NVIDIA Cosmos3-Nano-Sim-Auto (nvidia/Cosmos3-Nano-Sim-Auto), OpenMDW-1.1

MapMap is a member of the NVIDIA Inception program.

NVIDIA, the NVIDIA logo and NVIDIA Inception are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.

More MapMap demos on NVIDIA technology