We gave a small model an ontology. It mostly ignored it.

I keep hearing a version of the same idea in agent infrastructure:

Give the model the schema. Let it figure out which tool or entity type to use.

That sounds reasonable. Modern models are good at reading descriptions. Enterprise software already has schemas, APIs, tables, object types, and tool definitions. Put all of that in the prompt and let the model route.

I wanted to know how far that gets you with a tiny model.

The short answer: not very far.

We gave a 0.8B parameter Qwen model an enterprise-like catalog with 41 types. Random guessing gets 2.4% right. Giving the model the catalog in the prompt got 4.8% right on types it had seen during the dataset's training split.

Then we trained the same tiny model on the environment.

It jumped to 68% with ordinary fine-tuning, and 77.4% with the Infona version of the task.

That was the interesting result.

The more useful result came right after it: when we asked the model to handle completely new types it had never been trained on, it fell apart.

Good. Now we have something worth investigating.

The setup

We used Google's Schema-Guided Dialogue dataset, or SGD. It contains conversations across services and APIs in domains like travel, banking, restaurants, payments, events, and media.

We did not run the official SGD dialogue-state-tracking benchmark. We built a smaller task around the part of Infona I care about.

The task looks like this:

user request
    ↓
pick the right type from a local ontology
    ↓
run the skill attached to that type
    ↓
produce the structured arguments

Think of an internal company agent with a catalog like:

Customer
Order
Payment
Hotel
Flight
Restaurant
...

Each type has a small set of fields or actions associated with it.

For the experiment, we hid the helpful service and domain names from the model. Instead of telling it "Hotels_2" or "Restaurants_1," the catalog used opaque type IDs plus their available fields. The model had to infer which local type the user's request belonged to.

If it picked the wrong type, it didn't get to run some giant bag of tools and recover later. The wrong type meant the wrong skill.

That is much closer to the product question I care about: can a small model operate inside a customer's typed environment?

The main SGD test set we constructed had 8,208 examples across 41 types. Of those, 2,579 belonged to types represented during training. The remaining 5,629 belonged to services the model had not been trained on.

That distinction matters a lot.

First: just put the ontology in the prompt

This is the obvious baseline.

Take Qwen3.5-0.8B, give it the catalog, descriptions, and the user's request, then ask it to choose the correct type.

On the trained types:

That 4.8% is the number I keep coming back to.

The schema was there. The model could read it. It still barely knew what to do with it.

This is one experiment with one small model, so I don't want to turn it into a law of nature. But it is a useful warning against an assumption I see all over agent systems: making structure visible to a model does not mean the model knows how to operate inside that structure.

Training changed the behavior

Once we fine-tuned the 0.8B model on the local environment, things changed fast.

An ordinary fine-tune, with no special Infona machinery, reached 68.0% correct type selection on trained types.

Our Infona formulation reached 77.4%.

The Infona version trains around a typed contract: bind the request to the local type, then execute the skill associated with that type. On the same trained-type slice, it also reached a rescored slot F1 of about 0.74, versus 0.69 for the ordinary fine-tune.

I'm happy about the lift, but the control is important.

Most of the improvement came from training, not from some magical ontology trick.

If I only showed you 4.8% versus 77.4%, I could write a much punchier post. It would also be misleading.

The honest comparison is 68.0% versus 77.4% once both small models are actually trained. That's a 9.4 point improvement on the trained types. Interesting, yes. A new ML method, no.

Then the new types showed up

This is where the nice story broke.

The SGD test set intentionally includes services that are not present in training. That gave us a way to ask a harder question:

If I add a new type to the ontology, can the model use the schema and figure it out?

For our setup, basically no.

The Infona-trained model got 98 out of 5,629 unseen examples correct: 1.7%.

Random chance across the 41-type catalog is about 2.4%.

The ordinary fine-tuned model got zero unseen examples right.

Worse, the failures were not random. The model often collapsed a new type into a familiar sibling. A new restaurant service looked like the restaurant service it already knew. A new flight service got mapped to an old flight service. New hotel variants piled into trained hotel variants.

That sounds obvious when written out, but it is exactly the kind of failure that becomes expensive in production.

"Hotel" is not enough if one hotel API checks availability and another changes a reservation.

"Payment" is not enough if one tool refunds a charge and another opens a dispute.

The closer two skills look, the more you care about choosing the exact one.

So what did we actually learn?

I don't think the result is "small models need fine-tuning." We knew that was possible.

The useful part is the shape of the failure.

A local ontology can describe the world perfectly well and still be almost useless to a small base model if the model has not learned how that particular environment behaves.

After specialization, the same tiny model becomes surprisingly competent inside the part of the environment it knows.

Then you extend the environment and the competence disappears.

That makes me think about enterprise ontologies differently.

I've mostly thought about Infona as the layer that keeps facts typed, deduplicated, current, and cited for agents. But a typed environment can potentially do something else too: define the operating surface a small specialist model is trained to work inside.

The ontology stops being a giant reference manual stuffed into the context window. It becomes part of the model's local job description.

We're not there yet. The unseen-type result is the reason I'm writing this instead of pretending we solved it.

The next experiment

The next question is much more practical:

How expensive is it to teach the model one new capability?

Imagine the agent already knows a customer's Order type and these skills:

get_order
cancel_order
update_shipping_address

Tomorrow the company adds:

issue_partial_refund

I don't want the answer to be "retrain the whole thing on thousands of examples."

So the next benchmark will hold out entire skills, then introduce them with:

0 examples
5 examples
10 examples
25 examples
50 examples
100 examples

We'll measure how quickly the new skill becomes usable, whether the model starts confusing it with existing sibling skills, and whether teaching the new skill damages anything it already knew.

We'll start with public function-calling data, then move the result into a stateful enterprise-style environment where success means the tool call actually changes the underlying system correctly.

If a tiny model can absorb a new customer capability from a handful of examples without forgetting the old ones, that would be much more interesting than a routing benchmark.

If it can't, that's useful too.

A few important caveats

This was a constructed Infona task using SGD, not the official SGD benchmark. Don't compare these numbers to published dialogue-state-tracking or Joint Goal Accuracy results.

The 77.4% number is for trained/seen types only. Across the full 8,208-example test set, where 69% of examples came from unseen services, Infona's type-selection accuracy was 25.5%.

We have not shown that a 0.8B model can handle arbitrary new APIs. The current experiment says almost the opposite.

We also have not shown that Infona's training setup beats ordinary fine-tuning by a huge margin. On trained types, the difference was 9.4 points.

That's the whole point of keeping the ugly numbers in the post.

I want to know whether a company's ontology can become something models genuinely learn to operate inside, and whether that specialization can stay cheap as the company changes.

The first experiment says the prompt alone is not enough.

Now we get to test the harder part.