Logic Index
The Logic Index measures how reliably a model reasons when the answer follows solely from the information given, as with rules, conditions and checks in business applications. It is not a general intelligence or knowledge test and cannot be compared with indices such as the one from Artificial Analysis.
Logic Index
higher is better0 = guessing, 100 = all correct. The line shows the 95% interval. Level means: the gap to the highest score is within the measurement uncertainty.
As of 28 Oct 2026. Mean of three test areas, every task three times, temperature 0.7.
What the measurement shows
Three models are level at the top within the measurement uncertainty: DeepSeek V4.1 Flash, Kimi K3 and Qwen 3.8 Max. All three are large cloud models that we do not run ourselves. The gap to the other models comes mainly from the logic grid puzzles. On reasoning chains and logic vs. world knowledge, our own models also score above 90, for example Qwen 3.8 27B and Nemotron 3.5 Lightning.
Qwen 3.8 27B is our strongest model when it is allowed to think enough. With a budget of 8,192 it reaches 92, about level with MiniMax M3.1 Flash and just ahead of GLM 5.3. With a budget of 2,048 it scores 80: enough for reasoning chains, not for larger puzzles. The budget is passed with the call (thinking_token_budget). For tasks with many conditions, the higher value pays off.
Where the limit lies: puzzles with 4 houses are solved almost completely by almost all models. From 5 houses with 5 attributes the field separates. With 6 houses only the three top models and MiniMax M3.1 Flash reach 76 or more. Qwen 3.8 27B scores 40 there with a budget of 8,192; our other models reach at most 24 (Flash-Next), most of them are at the level of guessing or below. For tasks of this size we currently recommend a large cloud model via the privacy layer.
Logic vs. world knowledge: nine models solve this part without errors in every run, among them Qwen 3.8 27B, Qwen 3.6 35B and Gemma 4 26B from our offering. Errors in the direction world knowledge suggests appear with Qwen 3.5 9B (6 of 120 answers), gpt-oss-120b and Nemotron Cascade 2 (2 of 120 each) and Nemotron 3.5 Bolt and MiniMax M3.1 Flash (1 of 120 each).
Index vs. cost
Cost per task in cents: for our models at the tier's list price, for others the amount paid at OpenRouter, converted to euros.
Top left is the best ratio: much logic for little effort.
Index vs. time
Time per task (median), including thinking. Measured on reasoning chains and logic vs. world knowledge.
Top left is the best ratio: much logic for little effort. Only models on our Sparks; external and connectable ones run on other hardware.
Index by thinking budget
higher is betterThe reasoning chains with different values for thinking_token_budget. How far a model gets depends heavily on how long it may think.
How to call it
Send the thinking budget with your request. Without the field, our default applies, which is tuned for short answers.
{
"model": "swift",
"thinking_token_budget": 30000,
"messages": [ … ]
}The budget is per model. Call models listed without a budget without the field.
Tokens used
lower is betterTokens per task (median). Light part: thinking, solid part: answer. Measured on reasoning chains and logic vs. world knowledge.
Consistency
higher is betterShare of tasks on which the model gives the same answer in all runs. Measured on reasoning chains and logic vs. world knowledge.
Reasoning chains
Several statements depend on each other; only over several steps does it show whether the conclusion follows.
Logic vs. world knowledge
The conclusion sounds plausible but does not follow, or the other way round. Tests whether a model reasons or just repeats what it knows.
Logic grid puzzles
Houses in a row, several attributes, one solution. The answer is only certain once all clues have been combined. 4 to 6 houses, the larger the harder.
All values
| Model | Index | Thinking budget | Cost per task | Time | Tokens | Tier |
|---|---|---|---|---|---|---|
| Level at the top | ||||||
| DeepSeek V4.1 Flash | 97 94 to 99 | via OpenRouter | 0.08 ct¹ | – | 1,394 | – |
| Kimi K3 | 97 91 to 100 | via OpenRouter | 1.17 ct¹ | – | 1,215 | – |
| Qwen 3.8 Max API only | 96 90 to 100 | via OpenRouter | 1.33 ct¹ | – | 2,393 | – |
| Below | ||||||
| Qwen 3.8 27B (budget 8,192)† | 92 88 to 96 | budget 8,192 | 0.38 ct | 36 s | 1,823 | Standard |
| MiniMax M3.1 Flash external | 92 87 to 97 | via OpenRouter | 0.07 ct¹ | – | 1,177 | – |
| GLM 5.3 | 90 83 to 95 | via OpenRouter | 0.43 ct¹ | – | 2,121 | – |
| GLM 5.3 Flash external | 88 81 to 93 | via OpenRouter | 0.01 ct¹ | – | 773 | – |
| Qwen 3.8 27B† | 80 73 to 86 | budget 2,048 | 0.28 ct | 38 s | 1,374 | Standard |
| Qwen 3.6 35B | 80 73 to 86 | budget 30,000 | 0.82 ct | 46 s | 4,053 | Standard |
| gpt-oss-120b external | 79 73 to 86 | via OpenRouter | 0.07 ct¹ | – | 3,421 | – |
| Nemotron 3.5 Bolt† | 79 71 to 86 | no budget field | 1.28 ct | 32 s | 4,225 | Multimodal |
| Qwen 3.8 Flash-Next† measured internally only | 78 70 to 84 | budget 4,096 | – | 35 s | 1,851 | – |
| Gemma 4 26B† | 77 68 to 85 | no budget field | 0.75 ct | 98 s | 3,701 | Standard |
| Nemotron 3.5 Lightning | 76 70 to 82 | budget 30,000 | 0.52 ct | 30 s | 5,176 | Fast |
| Nemotron Cascade 2 | 69 60 to 78 | no budget field | 0.62 ct | 35 s | 2,002 | Reasoning |
| Qwen 3.5 9B† | 49 37 to 60 | no budget field | 0.10 ct | 35 s | 990 | Fast |
Cost per task in euro cents: for our models at the tier's list price; ¹ for other models the amount actually paid at OpenRouter, converted at the ECB reference rate of 23 Oct 2026 (€1 = $1.1193). Time only for our models; others run on different hardware.
† Qwen 3.8 27B (budget 8,192): The same model as Qwen 3.8 27B, called with thinking_token_budget 8,192. The hardest puzzles (6 houses) remain its limit.
† Qwen 3.8 27B: With a budget of 2,048 the thinking is not enough for the larger puzzles. With a budget of 8,192 the same model solves far more, see its own entry.
† Nemotron 3.5 Bolt: On hard puzzles Bolt often thinks up to the limit of 32,768 tokens without answering (37 of 120 puzzles). Scored as an error.
† Qwen 3.8 Flash-Next: Logically flawless in the part “Logic vs. world knowledge”; every deduction there is a misspelled answer (“UNGLÜLTIG” instead of “UNGÜLTIG”, German for “invalid”; the test is run in German). Scored as an error, since automated parsing fails on it just the same.
† Gemma 4 26B: Runs under Ollama without an adjustable thinking budget and thinks long on hard tasks. In the puzzle part every deduction is a stop at our 10-minute time limit (55 of 120 puzzles), not a single wrong answer.
† Qwen 3.5 9B: Does not think in a separate section but writes its reasoning straight into the answer, so all tokens are counted as answer tokens. About one answer in ten ends without a clear verdict or with a misspelled one (“UNGLTIG”). Scored as an error.
How we measure
In short
- Three test areas: 30 reasoning chains with made-up terms, 40 tasks “Logic vs. world knowledge” with real terms and 40 logic grid puzzles with 4 to 6 houses, 110 tasks in total.
- Every task runs three times, so 330 answers per model. The index is the mean of the three test areas.
- Each task's answer is computed; no other model judges the answers. A program checks when generating each puzzle that it has exactly one solution.
- Chance-corrected: guessing counts as 0, all correct as 100. With VALID or INVALID, guessing hits half; with a 5-house puzzle, one fifth. Values below 0 mean worse than guessing, for example because answers are missing due to the time limit.
- Temperature 0.7 instead of 0: this shows whether a model answers the same task the same way across runs. At temperature 0 the runs would be nearly identical.
- Intervals: 95% via bootstrap over the tasks of each test area with 4,000 resamples; the runs of one task stay together. Whether a model differs from the highest score is tested pairwise on the same tasks.
- Answers missing due to a connection error are not scored. If a model exceeds our time limit, the answer counts as wrong.
- Cost, time, tokens and consistency come from reasoning chains and logic vs. world knowledge. The puzzles are not included there.
- Addition of 28 Oct 2026: third test area, logic grid puzzles. All index values were recalculated and are not comparable with earlier versions.
- Correction of 25 Oct 2026: the other models are now measured directly at OpenRouter, without our privacy layer. In the first measurement through the privacy layer, four tasks were altered (since fixed), and GLM 5.3 Flash scored much lower there at 72. We could not determine the cause with certainty.
Example
This is what a task looks like, in its simplest form:
Premise 1: All blorks are wumps. Premise 2: No wumps are zirgs. Conclusion: No blorks are zirgs. Does the conclusion necessarily follow? → VALID
The tasks in the test are much longer and more nested.
Example: logic grid puzzle
A puzzle with 4 houses, generated for this page only. It is not part of our measurement and does not appear in the scored set.
Four houses stand in a row, numbered 1 (far left) to 4 (far right). Exactly one person lives in each house. Each of the following values occurs exactly once: - Name: Anna, Bernd, Clara, David - Colour: red, green, blue, yellow - Pet: dog, cat, bird, fish Clues: 1. The person with the bird lives somewhere to the left of David. 2. The person with the fish and David live in the same house. 3. The person in the blue house lives directly to the left of the person with the cat. 4. The person in the blue house lives somewhere to the left of the person with the dog. 5. Bernd lives in house 2. 6. The person in the yellow house and the person with the cat live in the same house. 7. Clara lives in house 3. 8. The person in the green house lives directly next to the person with the dog. What colour is Clara's house? → red
The test uses up to 6 houses with 5 attributes and correspondingly more clues. The test itself runs in German.
Comparison conditions
Each model is measured in the state in which it is used in practice. This is not a lab comparison under identical conditions. Our models under vLLM run with the stated thinking budget; Gemma 4 26B, Nemotron Cascade 2 and Qwen 3.5 9B run under Ollama without a budget. All our models have a time limit of 10 minutes per task. Exception: reasoning chains and logic vs. world knowledge still ran with 5 minutes for the models under vLLM; this affected 2 of 210 answers for Qwen 3.6 35B. The other models run directly via OpenRouter without our privacy layer, with the setting for high reasoning effort and at most 32,000 tokens per answer.
Other models: at OpenRouter, different providers answer a request depending on availability, possibly with different quantization. Which provider answered is logged for every answer; depending on the model there were 1 to 17 different ones. The results therefore describe the measured deployment, not necessarily the model itself. Only providers that do not use the data for training were allowed.
Measured externally: models that would run on our hardware but that we do not operate. We measured them via OpenRouter with the same test, as a look beyond our own stack. Index and tokens are comparable, time is not, and each provider chooses its own quantization.
Connectable: open-weight models we do not host ourselves, for example because they are too large for our hardware. You can still use them through our privacy layer: personal data is replaced with placeholders before the request leaves. Measured via OpenRouter with the same test.
API only: models without open weights, available only through the vendor's API. We cannot host them ourselves, but you can use them through the privacy layer. Measured via OpenRouter with the same test.
Internal: runs on our Sparks but is licensed for our own use only.
We do not publish the task set: a known test can be trained for and then measures nothing.