The Next Number Doesn't Have to Be 32

One of the first programs I ever wrote was called Animal.

It ran on a Research Machines 380Z, locked in a cupboard at school, somewhere in the geological strata of personal computing. I had a key. No cloud, no app store, no billions of parameters. Just a computer, some code and the intoxicating experience of making a machine do something because you had told it how.

Animal was wonderfully simple.

You thought of an animal. The computer asked questions.

Does it have feathers?

Does it live in water?

Is it bigger than a cat?

Eventually it guessed.

"A horse?"

No.

Then something interesting happened. Instead of merely failing, it asked what animal you had been thinking of and what question would distinguish it from the horse.

Suppose the answer was elephant.

"Does it have a trunk?"

Yes led to elephant. No led to horse.

The program inserted the new question into its decision tree.

Next time, it knew about elephants.

I knew there wasn't really a little zoologist emerging inside the 380Z. I'd written the program. Its entire understanding of the animal kingdom consisted of questions, answers and branches in a tree.

But it felt like learning.

And in a modest sense, it was.

What strikes me now is that Animal blurred something we tend to separate in modern AI.

Training and use were the same activity.

Every time someone played the game, they could alter its model of the world. Using it trained it. Training it improved future use.

The computer encountered something it didn't know, failed, asked what difference mattered and changed itself.

Nearly half a century later, surrounded by large language models that can write code, discuss Homer, analyse a business case and politely explain why my latest idea isn't quite as original as I hoped, I've started thinking about that little program again.

Not because ChatGPT is Animal with several trillion extra branches.

It isn't.

Because both raise a more fundamental question.

What does it mean to notice a difference?

My lazy way of solving sequences

I have another habit I've never regarded as particularly clever.

Give me a number-sequence puzzle and, if I can't immediately see the answer, I cheat.

Take:

2, 5, 10, 17, 26...

I subtract.

The differences are:

3, 5, 7, 9...

Now the next difference looks like 11.

So the answer is 37.

If that doesn't reveal anything, I take the differences between the differences.

For the sort of sequence somebody has deliberately designed as a puzzle, eventually something usually gives.

At the level where I was originally given the problem, I couldn't see the pattern.

So I changed the representation.

Instead of looking at the numbers, I looked at the relationships between them.

Suddenly the pattern became simpler.

That isn't what Animal was mathematically doing. It was building a decision tree, not calculating finite differences.

But both use the same trick.

Don't just look at the things.

Look at the differences between the things.

And that started me wondering about large language models.

Before we get carried away

A transformer isn't secretly constructing a difference table. ChatGPT doesn't have an enormous binary tree with DOG down one branch and ELEPHANT down another.

Modern neural networks learn distributed numerical representations across huge numbers of parameters. Their internal mechanisms are far more complicated than anything running on my 380Z.

So drawing a straight line from Animal through finite differences to GPT would be bad computer science.

The more useful connection is simpler.

Something which looks complicated in one representation can become predictable in another.

And prediction brings us to an old philosophical problem disguised as a school maths question.

Why is the answer 32?

Consider:

1, 2, 4, 8, 16...

What comes next?

32, obviously.

Except it isn't obvious.

There are infinitely many mathematical rules that fit those first five numbers and produce something other than 32 for the sixth.

We choose 32 because:

double the previous number

is such a wonderfully simple explanation.

Occam's razor slips quietly into the room.

Faced with many possible explanations, we prefer the one requiring the least machinery.

Double it.

And most of the time that's a very sensible thing to do.

But notice what happened.

We didn't deduce 32.

We chose the pattern we thought most plausibly generated the sequence, then continued it.

That's induction.

And suddenly the distance between my lazy number trick and a language model doesn't feel quite so ridiculous.

What comes next?

At heart, a language model is intensely interested in what comes next.

Given everything in the context so far, what continuation is most plausible?

"The cat sat on the..."

Mat.

"To be or not to..."

Be.

Easy enough.

Move into politics, project delivery, organisational behaviour or human relationships and the number of plausible continuations explodes.

The model has to navigate patterns of language, concepts, relationships, argument and context.

That can start to look very much like reasoning.

Sometimes it may effectively be reasoning.

But I still think we should separate three things.

Prediction: What probably comes next?

Reasoning: Why should it come next?

Verification: How do we establish whether it actually does?

Large language models are astonishing at the first and increasingly impressive at the second.

The third remains harder.

I've written elsewhere about the hallucination ceiling, particularly when we turn language models into agents. A chatbot producing a plausible wrong answer is irritating. An agent carrying that answer into another system and acting on it is a governance problem.

The research that prompted that piece argues from computational complexity that some tasks cannot simply be made reliably self-verifying by asking a transformer system to keep reasoning for longer.

The number sequence gives us a simpler version of the same problem.

Sometimes the most plausible next number isn't the true one.

Sometimes there isn't enough information to know.

Yet the game still asks for an answer.

Animal had an advantage

This is where my 380Z starts looking rather sophisticated.

When Animal didn't know, it admitted it.

More than that, it asked for help.

"What was your animal?"

"What question would distinguish it from mine?"

The failure became training data.

That is strikingly different from how we first encountered most modern language models.

Their training largely happened before we met them. Our conversation used the model but didn't normally rewrite its underlying weights there and then.

Animal said:

I don't know. Teach me the difference.

A language model is built to continue.

That is one reason hallucination is such an interesting word. We sometimes talk about it as though a normally truthful machine suddenly decides to lie.

Perhaps that's backwards.

The remarkable thing is that a machine designed to produce plausible continuations manages to be factually right so often.

The trouble begins when plausibility is mistaken for verification.

And that isn't only an AI problem.

Humans complete sequences too

We do it constantly.

"She resisted the last change, so..."

"Projects like this always..."

"Our best leaders usually..."

"Customers don't want..."

"He's not really management material."

Those are unfinished sequences.

We have observed something before, inferred a pattern and begun carrying it forwards.

Sometimes that's experience.

Sometimes wisdom.

Sometimes survivorship bias.

Sometimes prejudice.

The uncomfortable part is that all four can feel exactly the same from inside our own heads.

The pattern may even be statistically real.

Imagine an organisation whose senior leaders have historically come disproportionately from one kind of background.

An AI trained on previous appointments might identify the characteristics associated with promotion with impressive accuracy.

It hasn't necessarily hallucinated anything.

It may have found the historical pattern perfectly.

The mistake comes when we move silently from:

This is what happened

to:

Therefore this is what should happen next.

That distinction runs through much of what I've been writing about AI and organisations.

In In a World of Perfect Answers, What Still Stands Out?, I worried about AI producing polished answers while obscuring the judgement and accountability behind them. Elsewhere on eMote I've kept returning to the same territory: AI often doesn't create our organisational weaknesses so much as expose the ones already there.

This question of differences belongs there too.

Because organisations are enormous machines for creating categories.

Manager. Staff.

Permanent. Temporary.

High performer. Average performer.

Professional. Unprofessional.

Supporter. Resistor.

Some distinctions are essential.

Some are convenient.

Some stopped making sense years ago.

And some are historical prejudice fossilised into process.

A difference that makes a difference

Gregory Bateson's wonderful formulation was that information is a difference that makes a difference.

I've always liked that.

But Animal makes me read it slightly differently.

My little computer could ask for a difference.

It couldn't really decide whether the difference I gave it was a good one.

Suppose it was trying to distinguish a dog from a cat.

I could teach it:

"Does the English name contain the letter D?"

Perfectly effective.

Dog goes one way. Cat goes the other.

But it isn't much of a theory of zoology.

"Does it normally bark?" is better because the distinction captures something more useful.

The machine can store the distinction.

The human supplies its significance.

And that may be one of the most important questions we face as AI becomes embedded in organisations.

Not simply:

Can the machine find a difference?

It almost certainly can.

But:

Should that difference be allowed to matter?

That is a governance question.

It is also a DEI question.

And ultimately, it is a human judgement.

The tree was the knowledge

There is another reason I keep returning to that old program.

I could understand it.

Completely.

If Animal had decided dolphins were birds, I could inspect the tree, find the question sending it down the wrong branch and change it.

The structure was the knowledge.

Today's AI systems are incomparably more capable precisely because they aren't built from a few explicit branches I can print out and understand.

Their representations are distributed through huge neural networks.

We've gained extraordinary capability.

We've also lost something.

Transparency.

Asking the 380Z why it guessed elephant and I could retrace every question that got it there.

Ask a modern language model why it produced a particular answer and the explanation it gives you is itself another generated answer.

Useful, perhaps.

But not the same thing.

As our machines become better at finding patterns than we are, understanding which distinctions they are relying on becomes harder at precisely the moment those distinctions become more consequential.

Perhaps learning organisations have the same problem

We talk enthusiastically about becoming learning organisations.

But learning isn't automatically good.

An organisation can learn the wrong lesson with great efficiency.

Something goes badly.

We introduce another approval.

Something else goes badly.

We create another form.

Someone takes a risk.

We remove discretion.

A project succeeds in unusual circumstances.

We turn those circumstances into methodology.

The tree grows.

Every failure adds another branch.

Before long nobody can remember why half the questions are being asked.

My old Animal program would recognise the architecture immediately.

Perhaps organisational learning requires more than remembering distinctions.

It requires pruning the tree.

Ask:

What happened?

Then:

What keeps happening?

Then:

What pattern might explain it?

But don't stop there.

Ask the question my number sequences never require:

Should the pattern continue?

That's the bit Occam can't solve for us.

Nor can statistics.

Nor, ultimately, can an LLM.

Back to the elephant

I'm rather fond of the idea that one of the first programs I wrote might still have something to teach me.

At the time, Animal seemed clever because every mistake made it slightly better.

It encountered something new.

It asked for the missing distinction.

It changed its model.

Then it tried again.

Now we have machines which can find relationships across quantities of information that my 380Z couldn't have begun to store.

Yet I suspect the deeper challenge hasn't changed very much.

Learning involves noticing differences.

Intelligence involves finding patterns in those differences.

Prediction involves carrying the patterns forwards.

But judgement asks something else.

Is this a difference that should make a difference?

And sometimes the most intelligent thing we can do is look at:

1, 2, 4, 8, 16...

recognise exactly why everyone expects 32...

and still ask whether we have enough evidence to continue the sequence.