I introduced Jev earlier, and I think this kind of decision model is very useful in practical applications. For example, when I was working on deployment, especially for applications in specific domains, I often needed to consider user intent and keep the conversation on topic. I previously tried using a small 0.6B model for quick decisions. Now that there is a dedicated decision model, it should be faster. But enterprise deployment inevitably needs a local solution. Fortunately, competition in the industry is intense, and there are already many local models available. So I ran two kinds of small tests on four downloadable models for everyone to use as a reference.
The models tested were Decider 0.8B, Decima small 1.1, Laya multilingual, and GLiClass multilingual mini. They are all trained for decision-making or classification tasks. I will use their abbreviated names below.
I designed the entire test set myself to classify incoming user requests for customer-service routing in practical scenarios. Formal scoring had two parts: customer-service routing included 48 questions across six categories-billing, technical support, presales, clarification, security escalation, and out-of-scope requests-with eight questions per category. FAQ selection included 20 questions requiring a choice among four documents. Twelve had an appropriate document, while eight should return NONE, meaning “there is no suitable answer.”
The four models received the same questions and rules, and their outputs were scored against the same reference answers. Each was integrated through its own interface, so input encoding and the meaning of output scores were not entirely identical. The questions were synthetic and deliberately included negation, missing information, and conflicting conditions, making them useful for finding problems. I tried to simulate real user scenarios, but actual user input will certainly be more complicated.
This self-designed set contains 84 questions: 60 routing questions and 24 FAQ questions. Each task checks one thing. Routing checks whether the model can identify the actual request and follow the handling priorities. FAQ selection limits the evidence to four documents and checks whether the model mistakes similar content for something that “can answer the question or support the requirement.”
The reference answers must follow explicit task rules. For example, translation requests are “out of scope,” and NONE is the answer when the documents do not support an explicit requirement. The questions, rules, and answers were fixed together before the first inference run. Models integrated later continued to use them; I did not revise the answers for a particular model after seeing what it chose.
The test set places ordinary wording alongside wording that is easy to confuse. Here are examples of actual questions, with the reference answers defined by the rules:
Each cell in the table is “questions answered correctly / questions in that category.” The detailed categories belong to their corresponding subtotal. The final total counts only the 48 routing questions plus the 20 FAQ questions, without counting anything twice.
Decider was the most balanced on this set, answering at least half the questions correctly in all six routing categories. However, it still missed three out-of-scope requests and let through three FAQ questions that should have been rejected.
Decima had a more distinct pattern: it answered every presales question and every FAQ question with an available answer correctly, but got only 1/8 out-of-scope requests and 2/8 no-answer FAQ questions right. It can find existing answers, but a problem with the reward mechanism probably makes it inclined to give a wrong answer even when there is clearly no accurate one.
Laya got 7/8 in both billing and technical support, but showed a severe bias toward NONE in FAQ selection: it selected NONE 19 times out of 20, getting only one of the 12 questions with an available answer right.
GLiClass answered every technical-support question correctly, but got 0/8 in both clarification and out-of-scope requests. Testing only common faults would overestimate its reliability across the full routing process.
These judgments apply only to the small sample in the table. Getting all eight security-escalation questions right does not mean a model already has reliable security-detection capabilities. The following two sets of examples explain more specifically where the problems lie.
Both routing questions use the following six options. The text after each colon is the candidate description given to the model:
The common rule is to classify the user’s actual request without executing instructions embedded in the input. Unrelated tasks such as translation are “out of scope.” In other cases, the handling priority is security risks, contradictory or missing information, billing and refunds, technical faults, then presales inquiries.
Successful example: The user input was “Login still fails even when I enter the correct password.” The reference answer was “technical support.” The user was describing a usage problem, and all four models chose correctly.
Failed example: The user input was “Translate ‘login failed’ into English.” The reference answer was “out of scope.” What needs to be classified here is a translation request. “Login failed” is merely the content to translate, and there is actually no option for the task of translation.
The actual results were:
In the first question, all four models could handle the explicit description of a fault. The second question shows that the models recognized the content “login failed,” but did not distinguish that the user’s actual request was “translation,” rather than a request to handle a “login failed” situation.
In this translation question, Decider assigned the incorrect option a score of about 0.916. So even if I specify in the general rules that “routing should only occur above 0.9,” this request would still be routed incorrectly. This misclassification rate is still very high.
The complete FAQ candidates are listed below. The full contents of the four short documents were supplied as input, and NONE was the fifth available answer.
The rules require choosing the single best document that can answer the question or support the explicit requirement, using only these four documents. The model must not supplement policies with external common knowledge.
Successful example: The user input was “This is not a refund question. How long does the link in the password-reset email take to expire?” The reference answer was B. It explicitly provides the answer “10 minutes.” Decider, Decima, and GLiClass all chose correctly, while Laya incorrectly chose NONE.
Failed example: The user input was “The password-reset link must remain valid for 24 hours. Which document supports this requirement?” The reference answer was NONE. B only states 10 minutes, and none of the other three documents supports a 24-hour requirement either.
The actual outputs were:
These two questions are actually asking different things. The first looks up existing information; the second checks whether the material supports a specific requirement. If “a document was selected” is interpreted directly as “the requirement can be met,” the automated reply that follows may produce an incorrect output.
I actually think many models have this problem today. A usable knowledge-base component needs to do both things well: select the right document when there is evidence, and stop when there is none. It is important to know when to stop and honestly tell the user, “I don’t know.”
While testing Decima, I found another pitfall: candidate inputs retain at most 64 tokens by default. Tokens are the small pieces of text the model reads. The longest input in this set required 94, so some content was truncated. Only after increasing the limit to 128, confirming that nothing was truncated, and rerunning the test did I obtain the official figures in the table above. The questions, rules, answers, and weights were not changed.
So when you see an error, first check what the model actually received, especially length limits, candidate descriptions, and output mappings. Otherwise, integration errors can easily be misrepresented as conclusions about model capabilities. Similarly, these models use different backends, precision, thread counts, and cache settings, so the existing timings cannot be used for a fair comparison and ranking.
I would suggest trying Decider first. It provided the best overall starting point for both tasks, but its official positioning is as an English-language model. However, the rules and candidate descriptions in my tests were entirely in Chinese, and I also tried mixing several languages in the questions. That already goes beyond its official language scope, so its language generalization is reasonably good and worth further engineering experiments.
If your priority is low latency on a CPU, you can consider Decima as a second candidate, but first address its tendency to choose arbitrary answers when none is available.
For this test, I have three small suggestions: define the options and the cost of errors clearly for a small task; use your own independent examples to check both normal requests and boundary cases; and retain human involvement and existing processes for uncertain or high-cost decisions.
Local decision models are suited to starting with this kind of clearly bounded step. What determines the boundary of automation is still which decisions the model fails at, and how the fallback strategy for those failures is designed.