Models

Benchmarks

How Tase models are measured on tool calling, and what the current numbers are.

Tase models are built for one job: turning a sentence into the right tool call with the right arguments. General language benchmarks say little about that, so we measure it directly on a held-out set of real requests.

What we measure

Tool selection — the set of tools the model calls matches the expected set exactly. Order does not matter.
Arguments — every expected argument is present with the correct value. Extra arguments are tolerated.
False-call rate — how often the model calls a tool on a request that needs no tool at all. For an assistant with side effects, this is the metric that hurts users most.
Missed-call rate — how often the model answers in prose when a tool was required.

Method

The evaluation set is frozen: it is written once and never regenerated, so numbers stay comparable across model versions.
Held-out examples never appear in training data.
Tool schemas are not sent in the request for Tase models; baselines receive them in the prompt.
The model is called directly rather than through a serving layer, so nothing between the model and the score can alter the output.
Greedy decoding (temperature 0) for reproducibility.

Result schema

Every published number is generated from a machine-readable result record, not typed by hand. The record format is versioned, so a result can always be traced back to the model, dataset, and settings that produced it.

Results

Evaluation set: 60 held-out requests covering all built-in tools, in Turkish and English.

tase-0.1-8b — tool selection 48%, arguments 27%, false-call rate 0%.
Qwen3-8B (untuned, schemas in prompt) — tool selection 86%, arguments 50%, false-call rate 0%.

Tool selection accuracy across models — Picking the right tool is largely solved at this scale, provided the model can see the schemas.

Argument accuracy across models — Filling arguments correctly is where every model we tested struggles. This is the gap worth closing.

False-call rate across models — Calling a tool when none was needed. For an assistant with side effects, this is the failure users feel most.

What we learned

We built Tase 0.1 8B to test one idea: train the tool schemas into the weights so no schema has to be sent with a request. The benchmark did not support it. A general open model that simply receives the schemas in the prompt selects tools far more reliably, and it does not degrade as the tool list grows.

The more useful finding is what every model got wrong. Argument accuracy sits far below tool selection across the board: the model names the right function and then fills a date, an amount, or an identifier incorrectly. That is the gap our next model targets, and the benchmark now exists to prove whether we closed it.

Research preview

Tase 0.1 8B is published as a measured record of an approach we tested and set aside, not as a production default. Numbers appear exactly as measured, including where our own model loses.

How the field compares

We publish only numbers we measure ourselves, so we do not run GPT, Claude, or Mistral against our own tool set — a self-run comparison invites bias. Instead, here is the independent public leaderboard for this exact skill, with the open model Tase runs on placed among the rest.

Model	Function callingnon-live AST	Live functionslive AST	Knows when not to callirrelevance	Multi-turn	Overall
Tase 0.1 8Bruns on Qwen3-8B	87.58%	80.53%	79.07%	41.75%	42.57%
Claude Opus 4.5Anthropic	88.58%	79.79%	84.72%	68.38%	77.47%
Qwen3-8BQwen	87.58%	80.53%	79.07%	41.75%	42.57%
Claude Haiku 4.5Anthropic	86.5%	78.68%	85.11%	53.62%	68.7%
Mistral LargeMistral AI	84.65%	81.87%	68.92%	14.12%	38.37%
Llama 3.1 8BMeta	84%	70.76%	42.7%	11.12%	25.83%
GPT-4.1 miniOpenAI	83.83%	68.84%	81.69%	34.13%	50.45%
GPT-4.1OpenAI	82.79%	69.95%	86.52%	38.88%	53.96%
GPT-5.2OpenAI	81.85%	70.39%	79.42%	28.12%	55.87%
Mistral SmallMistral AI	73.6%	77.28%	87.94%	11.5%	37.15%

All figures published by Berkeley Function Calling Leaderboard (BFCL) v4 — an independent, public leaderboard for exactly this skill. This is a different test set from the Tase results above and the two are not numerically comparable.

† Tase 0.1 8B is a fine-tune of Qwen3-8B (Apache-2.0), and tool selection in the Tase API is carried by that base model with schemas supplied in the prompt. The row shows Berkeley's published scores for Qwen3-8B — Tase has not been submitted to BFCL as a separate entry. Tase's own fine-tune is measured on our internal set above, where it scores below the base model. Source: Berkeley Function Calling Leaderboard (BFCL) v4, retrieved 2026-07-21, evaluation commit f7cf735.

Reproducing

The benchmark runs against the same public API surface documented here. Send a request with no tools array and compare the returned tool_calls against your own expected output.

curl https://api.tase.app/v1/chat/completions \
  -H "Authorization: Bearer $TASE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tase-0.1-8b",
    "messages": [{"role": "user", "content": "Set an alarm for 7am"}],
    "temperature": 0
  }'

Pricing and limits Chat completions