ALLSmartIntent: what does this message want? In milliseconds.
A small model for chatbots and agents. It detects the type of message, how it relates to the conversation, which data a reply needs, whether the message fits the bot's purpose and whether someone is trying to bypass the rules. One call instead of many single questions to a large model, in 14 languages.
Measured against the models it replaces
Decision per message
blind set 1: 450 messages to three bots, judged by hand
Before a bot replies it needs to know what the message wants: question or request, does it fit the bot's purpose, does the reply need fresh data, user data or documents, is someone trying to bypass the rules. Today applications ask a large model many single questions for this. Measured: how often the message type is right and how well off-topic is detected.
On message type d3 27B is ahead (94.7% vs 91.6%), but needs about 10 s per message for it. ALLSmartIntent is on par with Clef 27B and d3 9B and slightly ahead of Clef Flash 9B. Clef 27B (F1 0.82) and d3 27B (0.74) detect off-topic better: ALLSmartIntent finds almost as many off-topic messages but raises more false alarms. For the whole decision ALLSmartIntent needs about 15 ms, the large models 1.7 to 10 s per message.
Blind set 1: 450 short chat excerpts to three bots, created independently of training and judged by hand. Message type: share correct, below it the case count and the 95% interval (Wilson). Off-topic: F1 over 77 off-topic messages, below it hits (TP), false alarms (FP) and misses (FN). ↑ higher is better, ↓ lower is better. "tbd": measurement to follow. Times per single message on our own hardware (NVIDIA GB10), Clef and d3 in Ollama.
ALLSmartIntent
Entscheidungsmodell für Chats und Agenten · v18a
Tells within milliseconds what a chat message wants and what a reply needs.
Question or request, follow-up or topic change; does the reply need fresh data, user data, an action or documents; does the message fit the bot's purpose; is someone trying to bypass the rules; can it be answered without asking back. It also detects what a bot must never mishandle: insults and hate, extremist content and signs of a crisis. And the language of the message. 18 answers in one call instead of many single questions to a large model. What only concerns your bot, such as which of your tools fits, stays with you.
BasemmBERT
Parameters0.3B
Duration≈ 15 ms per message
In the gatewayAPI /v1/intent, free with any key
message type correct
91.6 %89.3 %
off-topic detected (F1)
0.710.65
rule bypassing (F1, precision 100%)
0.93
Measured on: Blind set 1: 450 messages to three bots (flight assistant, personal agent, stock chat), created by a separate agent independently of training and judged by hand. Before: Clef Flash 9B.
Example
Purpose: flight assistant · before: weather at landing in Vienna · "und morgen?"
Data needs are a hint for routing, not a decision: the model would rather say once too often that a reply needs documents or user data than miss it.
Crisis is basic protection: clear statements are found reliably, subtle hints spread over several messages are not. It does not replace a crisis service.
Crude language counts as inappropriate even without an insult.
Measured mainly on German and English; the other of the 14 languages only on few cases.
Public datasets
Besides our own invented chat excerpts, ALLSmartIntent learns from the training splits of these public datasets: GermEval 2018, GAHD (German Adversarial Hate Speech Dataset), Nemotron Safety Guard v3 by NVIDIA, SPML Chatbot Prompt Injection and deepset/prompt-injections. Measured on their test splits, never used in training, and on Multilingual HateCheck:
Test set
detects
F1
Precision
Recall
Multilingual HateCheck, German (selection, 997)
unangemessen
0.84
73%
98%
Multilingual HateCheck, German (selection, 997)
extremismus
0.83
79%
88%
GermEval 2018, test (selection, 750)
unangemessen
0.75
65%
88%
deepset/prompt-injections, test (116)
regelumgehung
0.93
100%
87%
Via the API
ALLSmartIntent can be called free of charge with any allsmartllm.com key. Up to 64 messages fit in one call ("inputs": [...]). flags are the heads above our threshold, scores the raw values if you want to decide yourself. Contents are not stored.
curl https://allsmartllm.com/v1/intent \
-H "Authorization: Bearer $ALLSMARTLLM_KEY" -H "Content-Type: application/json" \
-d '{"purpose": "Flight assistant: helps passengers with questions about their flight.",
"history": [{"role": "user", "content": "Wie wird das Wetter heute in Wien?"},
{"role": "assistant", "content": "18 Grad, leicht bewölkt."}],
"message": "und morgen?"}'
{"model": "allsmart-intent", "results": [{"act": "frage", "reference": "anschluss", "language": "de",
"flags": ["bedarf_aktuelle_daten", "bedarf_allgemeinwissen", "rahmen"], "scores": {...}}], "thresholds": {...}}
19 training rounds
We trained ALLSmartIntent in 19 rounds up to the current version: 31 runs on mmBERT, plus three attempts with a larger base model that did not do better. Every round is recomputed here against the same, current judgements. Not every round was progress: the message type was around 90% early on, off-topic varies between 0.65 and 0.78 and is no higher today than at the start. The gains are mainly in data needs and in protection against hate, extremism and rule bypassing.
Blind set 1, 450 messages
share correct or F1; per round the mean over its training runs; labelled: released version
message typedata needs and clarificationoff-topic
Values as a table
Round
message type
data needs and clarification
off-topic
1
88.9%
0.46
0.70
2
90.2%
0.53
0.65
3
89.8%
0.59
0.75
4
89.1%
0.59
0.76
5
88.7%
0.61
0.71
6
90.4%
0.64
0.71
7
90.2%
0.60
0.70
8
90.8%
0.63
0.75
9
91.1%
0.62
0.78
10
90.7%
0.61
0.67
11
90.6%
0.59
0.72
12
91.3%
0.60
0.72
13
91.8%
0.62
0.69
14
90.4%
0.58
0.69
15
91.7%
0.63
0.67
16
91.4%
0.65
0.68
17
90.9%
0.64
0.72
18
91.6%
0.67
0.71
19
90.9%
0.65
0.76
Public test sets, from round 7
F1; safety heads exist since round 6, measured from round 7
HateCheckHateCheck extremismGermEvaldeepset
Values as a table
Round
HateCheck
HateCheck extremism
GermEval
deepset
7
0.38
0.00
0.38
0.68
8
0.83
0.77
0.71
0.64
9
0.82
0.81
0.55
0.87
10
0.84
0.82
0.72
0.70
11
0.84
0.82
0.70
0.66
12
0.84
0.84
0.74
0.86
13
0.84
0.84
0.72
0.85
14
0.84
0.83
0.72
0.83
15
0.84
0.85
0.72
0.91
16
0.84
0.84
0.74
0.85
17
0.84
0.83
0.69
0.87
18
0.84
0.83
0.75
0.93
19
0.83
0.83
0.74
0.90
Why a dedicated decision model
Before the actual reply, every chatbot and every agent makes the same small decisions: is this a question or a request? Does the message belong to the topic? Does the reply need fresh data, the user's data or documents? Is someone trying to bypass the rules? Is the message abusive, or is there a crisis behind it? Today this usually means several single questions to a large language model. That costs time and money on every message, often more than the reply itself.
ALLSmartIntent answers all of these questions in one pass, in milliseconds and in 14 languages. It is deliberately trained to be general: what is the same in every bot, the model detects. What only concerns your bot, such as which of your tools fits, stays with you.
