Which vision model reads a substation drawing best?
Loading results…
Every run: accuracy against output tokens
One dot per run. Color = model weights, shape = where it ran. Runs that returned no usable answer sit below the axis.
Three findings
Finding 1
Image resolution matters most
Finding 2
Same Nemotron weights: fewer failed runs on NVIDIA's endpoint
Finding 3
Nemotron spends far more output tokens, and "low" effort didn't lower them
| Setting sent | Reasoning tokens |
|---|---|
| none (default) | 563 |
reasoning_effort: "low" | 664 |
reasoning_effort: "high" | 821 |
enable_thinking: false | 0 |
Time and cost per run
Time is wall-clock per run, including a retry when the first reply was rejected. NVIDIA API catalog (hosted) runs used trial credits, so no price is shown.
What I'd ask NVIDIA
- Honor
reasoning_efforton the NVIDIA API catalog (hosted), or document the supported way to cap reasoning. "low" produced more reasoning tokens than the default in my text test (n=1 per setting), and on drawings some replies used the whole output budget on reasoning and returned no answer. - Document the input resolution Nemotron VL expects: native size, whether large images are tiled or downscaled, and what to send for wide engineering drawings. The same drawing scored very differently at 1024 px and 1642 px.
How this was measured
- A synthetic 735/315 kV single-line diagram with a hand-checked answer key: busbars, equipment, transformers, connections, title block, notes and external connections (66 items).
- Each run asks the model for JSON in a fixed schema at temperature 0, checks it (schema, then wiring rules), and gives it one retry with the exact error. Then the code builds an IEC 61850 SCL file and validates it against IEC's official XSD.
- Scores count items that exactly match the answer key. Items the key doesn't list are counted as extras, not errors.