Maybe the Model Isn't the Interesting Part Anymore

I've been playing with an autonomous AI experiment called Lee.

The original question was pretty simple:

Can I make a local LLM do something interesting on its own?

Lee has gradually become more complicated. It researches topics, searches for information, analyzes what it finds, writes reports, publishes them, remembers previous research, classifies the kind of evidence it found, and—when things go wrong—can record that failure rather than pretending everything worked.

I recently came across something from NVIDIA that made me look at the experiment a little differently.

NVIDIA demonstrated an agent architecture built around Claude Opus 5 that reportedly took a system scoring around 30% on ARC-AGI-3 and pushed it to 100% through the use of an agent "harness."

And that's the part that caught my attention.

Not because I think Lee is remotely comparable to NVIDIA's system.

It isn't.

But because the underlying idea is remarkably similar to what I've been stumbling toward with Lee.

The Model Isn't the Whole Agent

We tend to talk about AI systems as though the model is the agent.

Claude is the agent.

Gemini is the agent.

GPT is the agent.

Gemma is the agent.

But that's increasingly not how these systems actually work.

The model is more like the brain sitting inside a larger machine.

The surrounding system determines what information reaches the model, what tools it can use, what it remembers, what happens when it gets stuck, what gets fed back to it, and what happens with its output.

That's the "harness."

And NVIDIA's work is a pretty dramatic demonstration of how much difference that surrounding architecture can make.

Which got me thinking about Lee.

Lee Has Been Becoming a Harness Experiment

When I started this, I was mostly interested in the model.

Could a relatively small local model produce interesting writing?

Then I added a research pipeline.

Then memory.

Then research history.

Then different models for different stages.

Then quality evaluation.

Then epistemic classifications.

Then methodological caveats.

Then failure tracking.

At first, these looked like individual features.

But looking at NVIDIA's result, I think they're actually pieces of the same thing.

They're the harness.

Lee's underlying models are only one component.

The rest of the system determines how those models behave.

For example, Lee doesn't simply receive:

Research this topic and write an article.

The system can instead establish what kind of research it's supposed to perform, discover sources, ingest evidence, analyze it, distinguish different kinds of knowledge, remember what it has already researched, evaluate the result, and record what happened.

That is a very different proposition.

The Interesting Question Has Changed

Originally, my experiment was essentially:

"How good can I make a local AI?"

I think the more interesting question now is:

"How much useful behavior can I manufacture through architecture around a relatively modest AI model?"

That's a much more interesting experiment.

Because it changes what I should be measuring.

Instead of simply asking whether Gemma or DeepSeek or whatever model I'm using is "smart enough," I can ask:

• How much does memory help?
• How much does specialization help?
• Does giving different stages different models improve the result?
• Does explicit epistemic classification reduce hallucinated certainty?
• Does failure detection improve the system more than simply switching to a larger model?
• Does persistent research history make later work better?
• What happens when the agent is allowed to recognize that it doesn't have enough evidence?

Those are architectural questions.

And they're questions I can actually experiment with on my own hardware.

The "Research Failure" Experiment Suddenly Makes More Sense

One of the things I've recently added to Lee is the ability to recognize a research failure.

That sounds almost trivial.

If the research doesn't work, mark it as failed.

But there's something important about that.

A traditional AI workflow is often optimized around producing an answer.

The system has an incentive to produce something.

An autonomous research system needs a different capability:

knowing when it didn't actually accomplish the task.

That's part of the harness too.

If Lee researches a topic and the evidence doesn't support the requested conclusion, the correct behavior isn't necessarily to write a confident article anyway.

Sometimes the correct output is:

Insufficient Evidence.

Sometimes it's:

Research Failure.

Sometimes the topic turns out to be a different kind of research than originally assumed.

Those distinctions are becoming part of the architecture.

This Also Changes How I Think About Local Models

This is probably the most interesting consequence for me.

I've spent quite a bit of time thinking about whether Lee needs a larger model.

That's still a legitimate question.

But I'm increasingly convinced that simply throwing a larger model at the problem isn't necessarily the most interesting next step.

If the experiment is partly about agent architecture, then a 16 GB GPU isn't necessarily a dead end.

I can use different models for different jobs.

I can improve the memory system.

I can improve retrieval.

I can improve feedback.

I can improve failure detection.

I can improve the research contract.

I can improve the supervisor logic.

And I can measure what each change actually does.

In other words, instead of asking:

"How do I get a smarter brain?"

I can ask:

"How do I build a better nervous system around the brain I already have?"

That's a Different Kind of AI Experiment

And honestly, that's where Lee has become more interesting to me.

I started this as an experiment in local AI.

It may be turning into an experiment in agent architecture.

The model still matters. A bad model can't magically become a great one because you wrapped it in some clever Python.

But once the model is capable enough, the architecture around it may become increasingly important.

That's exactly what NVIDIA's demonstration made me think about.

The headline is essentially:

Maybe the harness is becoming as important as the model.

And if that's true, I've accidentally been building a pretty good little laboratory for testing the idea.

Lee isn't Claude.

Lee isn't NVIDIA's system.

Lee isn't going to score 100% on ARC-AGI-3.

That's not the point.

The point is that I can keep changing the machinery around Lee and see what happens.

And that's actually a much more interesting experiment than simply downloading a bigger model.

Maybe the next breakthrough for Lee won't be a smarter model.

Maybe it'll be a better harness.