Benchmarks
How Tase models are measured on tool calling, and what the current numbers are.
Tase models are built for one job: turning a sentence into the right tool call with the right arguments. General language benchmarks say little about that, so we measure it directly on a held-out set of real requests.
What we measure
- Tool selection — the set of tools the model calls matches the expected set exactly. Order does not matter.
- Arguments — every expected argument is present with the correct value. Extra arguments are tolerated.
- False-call rate — how often the model calls a tool on a request that needs no tool at all. For an assistant with side effects, this is the metric that hurts users most.
- Missed-call rate — how often the model answers in prose when a tool was required.
Method
- The evaluation set is frozen: it is written once and never regenerated, so numbers stay comparable across model versions.
- Held-out examples never appear in training data.
- Tool schemas are not sent in the request for Tase models; baselines receive them in the prompt.
- The model is called directly rather than through a serving layer, so nothing between the model and the score can alter the output.
- Greedy decoding (temperature 0) for reproducibility.
Result schema
Results
Evaluation set: 60 held-out requests covering all built-in tools, in Turkish and English.
- tase-0.1-8b — tool selection 86%, arguments 50%, false-call rate 0%.
What we learned
Tase 8B carries the tool schemas in its weights, so a request never ships a schema payload — smaller requests, lower latency. On the internal set it picks the right tool 86% of the time and never fires a tool on a plain conversational turn.
When it is wrong, it is almost always the arguments, not the tool name: the model picks the right function and then fills a date, an amount, or an identifier incorrectly. That is the gap the next model targets, and this benchmark exists to prove whether we closed it.
Built for the Tase app
How the field compares
We measure only our own model on the internal set above — running rivals ourselves would invite bias. For the broader field, here is the independent public Berkeley leaderboard for this exact skill, with Tase 8B placed against frontier and open peers.
| Model | Function callingnon-live AST | Live functionslive AST | Knows when not to callirrelevance | Multi-turn | Overall |
|---|---|---|---|---|---|
| Tase 8B | 87.58% | 80.53% | 79.07% | 41.75% | 42.57% |
| Claude Opus 4.5Anthropic | 88.58% | 79.79% | 84.72% | 68.38% | 77.47% |
| Claude Haiku 4.5Anthropic | 86.5% | 78.68% | 85.11% | 53.62% | 68.7% |
| Gemini 2.5 FlashGoogle | 84.96% | 74.39% | 93.67% | 36.25% | 56.24% |
| Llama 3.1 8BMeta | 84% | 70.76% | 42.7% | 11.12% | 25.83% |
| GPT-4.1 miniOpenAI | 83.83% | 68.84% | 81.69% | 34.13% | 50.45% |
| Mistral SmallMistral AI | 73.6% | 77.28% | 87.94% | 11.5% | 37.15% |
† BFCL figures reflect the open 8B model Tase is built on; Tase itself is measured on the internal set above. Tase has not been submitted to BFCL as a separate entry. Source: Berkeley Function Calling Leaderboard (BFCL) v4, retrieved 2026-07-21, evaluation commit f7cf735.
Reproducing
The benchmark runs against the same public API surface documented here. Send a request with no tools array and compare the returned tool_calls against your own expected output.
curl https://api.tase.app/v1/chat/completions \
-H "Authorization: Bearer $TASE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "tase-0.1-8b",
"messages": [{"role": "user", "content": "Set an alarm for 7am"}],
"temperature": 0
}'