Wakeline
Research · Learning in Worlds

Language as a second view of a learned world

A continual learning system that learns objects, words and their use from raw pixel and sound streams

Nov 3, 2026 · 44 min read

Abstract. We tested whether the learning layer of our Continual Learning System (CLS) can learn language the way a toddler does, from seeing and hearing, without text corpora, pretraining or gradient descent. In a small synthetic world it receives only the brightness of each pixel and a raw sound stream. From these it learns the objects, finds the words, binds each word to its object, and reads who did what to whom from timing. It then uses words to act. It asks for food with a word it has only ever heard (first-try accuracy 0.94, against 0.10 for a learner that learns words by trial and error), says what happened in an event it watched, relies on a second machine with which it shares only words, and learns which partner to trust. Without a narrator, two machines invent a shared vocabulary of their own. Reciprocity did not appear: when a gift cost the giver energy and could pay off only by being fed in return later, the machines did not learn to take turns giving. Of 136 pre-registered predictions, 114 held, 4 held in part and 18 failed.


What we wanted to find out

Our working hypothesis at Wakeline is that a learner's environment carries enough information for it to build, quickly, the structure it needs. One question we care about in particular is how language can be learned without large text corpora, the way a toddler learns it: by seeing things, hearing words while they happen, and working out which belongs to which. In that view a word is a second way to reach something the learner already knows from seeing it, and a sentence is a second way to reach an event.

We wanted to know how far the learning layer of our Continual Learning System (CLS) carries along this path. We asked it to do every step: to find objects in raw pixels, to find words in a raw sound stream, to bind the words to the objects, to read word order as who did what to whom, and finally to use words to ask for things, to describe what it saw, and to communicate with a second copy of itself. The CLS learns incrementally from what it observes, without pretraining or gradient descent, and what it learns can be inspected directly.

Relation to our gSCAN and ReaSCAN results

In a separate report, Learning to follow commands from demonstrations, the same learning layer is evaluated on gSCAN and ReaSCAN, two public benchmarks of grounded language. There it learns from demonstrations to carry out commands in a grid world, and it reaches 100 % exact match on every split of both benchmarks.

The benchmarks test one part of what this report is about. Their world is given as symbols: each object arrives as its shape, color, size and position, so there is nothing to see. Commands arrive as written words, already separated, so there is nothing to hear and nothing to segment. Language also flows one way. The learner receives a command and carries it out; it never needs anything, never speaks, and never has to make itself understood by another learner. How objects and words are found in raw signals, and how a learner uses words to get what it needs, lie outside what these benchmarks can test.

For the benchmarks, we switched off the part of the system that learns from raw signals: the perception layer that forms object classes from pixel brightness, and the segmentation of words from sound. In their place, the higher learning layer receives the benchmark's symbolic object attributes and the words of the command. The learning layer itself was not changed, and no new learning algorithm was written for gSCAN or ReaSCAN. Which word refers to which attribute, the agent's own frame of reference, the step-by-step navigation and the action patterns of adverbs are all learned by the same learning layer, from the benchmarks' demonstrations. What was built for the benchmarks is the connection to them: the symbolic input, step-by-step execution in the benchmark's grid, the kinds of situation each part of the learner tells apart, a preference for the most specific situation that applies, and, for ReaSCAN's nested clauses, a locality prior. The gSCAN report lists all of it.

What we did

We built a small synthetic world. Objects of five kinds move across a gray-level pixel field (16 × 16, 12 × 48 or 32 × 28 pixels, depending on the step); each kind has its own brightness pattern. A speaker produces synthetic syllables while things happen: it names one of the objects on screen, or it narrates an event such as one object hitting another. The CLS receives the brightness of each pixel at each frame and, separately, the sound as a single stream of samples. It is told nothing about where objects are, how many kinds exist, where one word ends and the next begins, or which word belongs to which object.

The same learner processes both streams. It learns object classes from the pixels and word forms from the sound, and it learns which words go with which object classes from what it sees and hears at about the same time. Appendix A summarizes what the system learns.

We then ran a sequence of steps, each adding one capability on top of the previous ones: objects, words, naming, word order, asking, describing, two machines relying on each other, feeding each other at a cost and trusting a partner, a second verb, and inventing words without a narrator. Each step had its protocol, its predictions and its refutation criteria committed to version control before the scored run. Each was run on three seeds, with controls that take away the piece under test and should fail if the result came from something else. The pre-registered predictions are graded in Appendix B, including the ones that failed.

Video 1: The learner in four acts, from scored runs (3 min, with the syllables played as tones). Seeing and naming come from a naming run with all four perception additions, at pixel noise σ = 0.02 (seed 0), who did what from the order test with token words (seed 0), and asking and the two machines from seed 2. In every act the words are syllable tokens cut by a simple segmenter. "What it sees" shows what the learner recognizes in each frame, in the color of the learned class; a class takes its word's color once a word is bound to it. Every scene shown plays in real time, and the learner learns from every scene, shown or cut.

The scenes in the video were picked by position: the first, middle and last scenes of a phase, and the first test scenes. There are two exceptions. The new-place scene is a shape's second pass through those rows, because first passes fail (see Limitations). The two-machine scenes come from seed 2, where both machines' first tries succeed, as they also do on seed 0.

What we found

Results are given as accuracy, the share of test cases the system gets right, from 0 (none) to 1 (all), unless another measure is named. Chance levels differ by test and are given in the tables.

StepWhat is testedMeasureCLSChance or comparison
Objects from pixelsTest sightings named as their own kindAccuracy1.00 (120 of 120)Chance 0.20
New placesA kind recognized in rows it never visitedAccuracy0.90 (54 of 60)0.00 without the third perception addition
Words from soundRecognized sound units that lie on their own wordPurity1.00No unit crosses a word boundary
NamingThe heard word picks the named object (in never-named places)Accuracy0.98 (0.96)Chance about 0.42 (0.40); words heard during the wrong scenes 0.56 (0.42)
Who did what to whom"A hits B" told from "B hits A"Accuracy0.90 to 1.00Chance 0.5; timing ignored 0.50 to 0.58
AskingThe new food asked for with a word never saidFirst-try accuracy0.94Trial and error 0.10; chance about 0.15
DescribingThe right words in the right orderAccuracy0.75 to 1.00Chance 0.05; same words in random order 0.39 to 0.50
Two machinesAsker says a never-said word; giver follows a never-asked wordFirst-try accuracy0.83; 0.89Trial and error 0.00; 0.47
TrustHonest creatures harmedShare of hungry episodes0.01Partner chosen at random 0.65
ReciprocityAsks answered with a gift, credit across episodesShare of asks0.05Each episode closed 0.06 (no reciprocity)
Second verbThe heard verb picks its kind of sceneAccuracy0.60 to 0.96Chance 0.5; verbs shuffled in training 0.40 to 0.68
Words without a narratorThe hungry machine gets what it needsSuccess rate0.99Trial and error 0.52; random speakers 0.14
Overview of the main results, three seeds pooled. Ranges run over test categories; the sections below give every category, control and condition.
  • Objects from pixels. The learner finds the five kinds of objects in raw pixels by itself and names 120 of 120 test sightings correctly, without ever mixing two kinds (purity 1.000: every learned class belongs to a single kind). It also recognizes a kind in rows where that kind has never been (54 of 60), and it keeps working under pixel noise up to σ = 0.02.
  • Words from sound. The same learner, on a single sound sensor, cuts an unbroken syllable stream into words. Every recognized sound unit lies on its own word (purity 1.00), and the share of units that end within one frame of the word's end is 0.97.
  • Naming. Words found in the sound bind to object classes found in the pixels from co-occurrence alone: the heard word picks the named object with accuracy 0.98, and 0.96 in places where that object was never named.
  • Who did what to whom. With every part learned from the raw streams, the system tells "A hits B" from "B hits A" with accuracy 0.90 to 1.00 (chance 0.5), including for two words it had only ever heard as names. No grammar rule and no role labels are given. Order is carried by timing: the object that acts, the agent, moves first, and its word is heard first.
  • Asking. A creature whose food changes asks for the new food with a word it has only ever heard and never said. Its first-try accuracy is 0.94; a creature that learns words by trial and error reaches 0.10.
  • Describing. Watching an event, the creature says what happened, "A hits B", with the right words in the right order: accuracy 1.00 on seen pairs and new roles, and 0.98 with two words it had only heard as names. With one new word next to a known one, accuracy is 0.75.
  • Two machines that share only words. Two copies of the system, each with its own perception, rely on each other on first tries: one asks with a word it has never said (first-try accuracy 0.83), the other answers a word it has never been asked with by handing over the object it names (0.89).
  • Trust, not reciprocity. Creatures that learn what each partner hands over in response to their words stop asking an exploiter: the share of hungry episodes in which they are harmed falls to 0.01, against 0.65 when choosing a partner at random. They drop a partner that turns after about two harmful gifts, against about ten by learned value alone. Giving at a cost did not become reciprocal: when a gift costs the giver energy and can pay off only by being fed in return later, the creatures did not learn to take turns giving. With credit reaching across episodes, the share of asks answered with a gift stays at 0.05, against 0.06 without it.
  • A second verb. Hearing "hits" or "follows" picks the matching kind of scene with accuracy 0.60 to 0.96 (chance 0.5), including pairs of shapes never seen with that verb.
  • Words without a narrator. Two machines that start with syllables and no words, with true object kinds in place of learned perception, settle on one word per kind and then get what they need with a success rate of 0.99. Each run invents a different vocabulary.
  • Predictions. Of 136 pre-registered predictions in the steps whose results are reported here, 114 held, 18 failed and 4 held only in part. Appendix B lists every failure.

The world and how it is scored

The pixel world has five kinds of objects. All objects of one kind look the same: a moving shape with the kind's own brightness pattern of nine samples, which differs from every other kind's. Two or three objects cross the field at once along separate lanes. In naming episodes the speaker names one of them. In event episodes one object, the agent (the one that acts), travels to another, the patient (the one acted on), and stops; the patient then moves away, and the speaker narrates "A hits B" as it happens: the agent's word while the agent moves, the verb at contact, the patient's word while the patient moves. During training every kind is named only in lanes that start in the first six rows or columns, so that naming can be tested in lanes where a kind was seen but never named. Two kinds are only ever named and never appear in an event during training, so that word order can be tested on words that never filled a role.

The speaker's syllables are synthetic waveforms of four to six samples, each with two pronunciations at slightly different levels, heard one sample per frame on a single sound sensor, with no pause inside an utterance. There are no word boundaries in the input.

All scores are shares of test episodes. A naming test counts the episodes in which the heard word picks the named object among the objects on screen. An order test plays the spoken sentence and its reversal (the same syllables with the two object words swapped) and counts how often the spoken one scores higher. The word–object associations are frozen during test blocks; perception keeps learning. Every number is pooled over three seeds unless stated.

Objects from pixels

The learner receives each pixel's brightness, frame by frame. From it, the learner forms object classes, without being told where objects are or how many kinds exist. For this world, four additions were made to the perception layer; Table 2 tests each of them.

MeasureResult
Objects found (recognized on the object)0.97
Purity (no class mixes two kinds)1.000
Test sightings named as their own kind120 / 120
Pairs of kinds sharing a main class0
Learned units per kind1–7
Table 1: Recognition by the learned object classes, one object at a time, three seeds, 150 training and 76 test episodes per seed. Chance for naming a kind is 0.20. With two or three objects on screen at once (the naming world, 60 test episodes per seed), purity measured per pixel is also 1.000.
ControlEffect
None of the four additions63 / 120 sightings named; one object crossing the grid falls into about eight classes
First addition, without its similarity checkpurity 0.70–0.90; two or three kinds merge in every seed
Second addition removedpurity per pixel 0.39–0.57 with several objects on screen
Third addition, without its similarity checkpurity 0.22–0.29
Table 2: Controls that remove all four additions, or remove or weaken one of them.

New places. Without help, a kind is recognized only at pixels that have seen it. With the third addition, which lets what is learned in one place be used in others, a kind is recognized in 54 of 60 sightings in rows it never visited, against 0 of 60 without it, with purity 1.000. Words bound to a kind elsewhere name it there correctly in 0.91 of tests with a word memory and 0.87 with the learner's own word–object associations (token words).

Noise. The pixel noise in this report is white: zero-mean Gaussian noise, drawn independently for every pixel and frame and clipped to the brightness range from 0 to 1; σ is its standard deviation on that scale. Without the fourth addition, any such noise stops recognition. With it, each pixel adapts to its own noise level, and recognition holds as in the noise-free world up to σ = 0.02: the kind is still named correctly, although at σ = 0.02 a kind's sightings fall on its single main class in 104 of 120 rather than 120, because noise splits some kinds over two to five classes. Without noise the addition changes nothing, bit for bit.

Pixel noise σ (0–1 scale)Kind namedPurityNever-visited rowsUnits stored, vs noise-free
0120 / 1201.00054 / 601.0×
0.005 to 0.02118–120 / 120≥ 0.99654 / 601.34–1.46×
0.0380 / 1200.42–0.82kinds merge (27 sightings named as another kind)1.26–1.37×
0.0524 / 1200.30–0.37kinds merge (49)1.11–1.15×
0.02 without the noise addition0 / 120–0 / 6013×
Table 3: Recognition under white pixel noise, three seeds pooled, with all four additions unless stated. Never-visited rows: sightings there that match the kind's main class.

At σ = 0.03 the tolerance the noise requires equals the distance between the two closest kinds in this world, and they merge. Both parts of the noise addition are needed: one alone recognizes nothing, and the other alone recognizes mostly background noise (63 % of recognitions).

Words from sound

The same learner, unchanged, runs on a single sensor that hears the syllable stream and cuts it into units. Syllable lengths vary so that words cannot line up with a fixed unit length by coincidence.

Measure200 naming episodes600 naming episodes
Words during which a learned unit is recognized1.001.00
Purity (recognitions on the unit's own word)1.001.00
Recognitions ending within one frame of the word's end0.970.98
Table 4: Segmentation of the sound stream by the learner, three seeds pooled.

No learned unit crosses a word boundary. Each word ends up with one or two units per surface form, of which a word has four.

Naming

The words found in the sound are bound to the object classes found in the pixels from what is heard and seen at about the same time. A heard word picks the object on screen it is most strongly associated with. Nothing tells the learner which sounds belong to which scene.

Arm200 naming episodes: named / never-named lanes600 naming episodes
Learned words, learned classes0.86 / 0.830.98 / 0.96
Same words, heard during the wrong scenes0.43 / 0.430.56 / 0.42
Table 5: Share of test episodes in which the heard word picks the named object, three seeds pooled. Chance is about 0.42 in named lanes and 0.40 in never-named lanes.

The word carries over to lanes where the object was never named, because it is bound to the object's class and not to a place. The timing between a word and its object is learned as well: in the same world with token words, the associations that last concentrate within about two frames of the moment a word ends.

Figure 1: The last naming scene of the run in Video 1, with pixel noise σ = 0.02. Left, the raw pixels the learner receives, drawn on a square-root gray scale so that the noise is visible. Middle, what the learner recognizes in this frame, in the color of its learned class, with the class's word above it. Right, the classes it has learned, gray until a word is bound to them. A white frame marks a class recognized in this frame; a frame in the word's color with ♪ marks the class recalled by the word just heard. The gray classes are extra classes into which noise splits some kinds, or partial views of an object. Words are shown as made-up syllables.

Who did what to whom

The learner responds to change: a resting object produces almost no activity, and a moving one a lot. In a "hits" event the agent moves first and the patient starts moving only at contact, so the agent's class becomes active before the patient's. The speaker narrates as it happens, so the agent's word is heard while the agent is active and the patient's word while the patient is. The listener is the naming listener, unchanged. The test asks which of "A hits B" and "B hits A" fits a scene. Nothing in the readout knows sentence positions or roles.

ArmPairs seen in trainingKnown words in new rolesOne new wordTwo words only ever named
Learned words, learned classes0.981.000.970.90
Same, with the true object kinds1.001.000.941.00
Timing ignored0.550.570.500.58
Sentence spoken after the event0.480.510.560.58
Table 6: Share of test events in which the spoken sentence beats its reversal, three seeds pooled, 0.5 being chance. Per seed: 18, 18, 12 and 20 test events. Field 12 × 48 pixels.
  • The category that matters most is the last one. Two kinds were only ever named in training, never heard in a sentence about an event. Their words still come out in the right order in 0.90 of tests, because each word binds to whatever is active when it is heard, and the agent is active first.
  • Timing carries it. With timing ignored, every category falls to chance.
  • Narration in the moment only. When the sentence is spoken after the event, accuracy is at chance, because words are bound only to what happens within a short window around them.
Figure 2: A test event from the order test with token words (Video 1, seed 0). Both object words had only been heard as names, never in a sentence about an event. The learner reads "bifo voda teno" as the yellow class hitting the green one, which is what happened; the tick means that the heard order scored higher than its reversal. Table 6 reports the same test with words found in the sound.

Asking for things

A creature that cannot move has an energy level that falls over time, and one object kind is food. A scripted caretaker brings the kind whose word the creature says. The creature learns which object class restores energy. To speak, it uses what it learned while listening: it says the word most strongly associated with the class it wants. Partway through, a second kind becomes food. The creature has heard that kind's word many times, but has never said it. A third kind, seen and heard as often but never food, is the control. In this step the words are clean syllable tokens cut by a simple statistical segmenter, and the objects are the learned classes.

ArmAsks for the new foodAsks for the control kindNew food among never-said asks
Learned classes, speaking from what it understands0.940.001.00
Same, true object kinds1.000.001.00
Same wants, shuffled word–object associations0.200.100.67
Trial and error (values each word said)0.100.110.47
Random speaker0.130.150.47
Table 7: First hungry requests after the food has changed, 100 probes per seed, three seeds pooled. Uniform chance of the new food is about 0.15.

The creature says a word it has only heard, to get what the word names, the first time it needs it. Understanding and speaking use the same store. Shuffling the associations removes the effect while leaving what the creature wants unchanged, so the word choice runs through comprehension. A creature that learns by trial and error has no preference between the new food's word and any other unsaid word, and reaches the new food only by exploring, after two to six episodes.

Figure 3: The first test episode after the food has changed (Video 1, seed 2). The creature wants the new food and says its word, "pasi", which it has heard many times but never said, and the caretaker brings it. The caretaker panel on the right shows the world, not what the creature knows.

Saying what happened

Here the creature watches one object hit another and says a three-word sentence. It hears and sees as in the order test, with words and object classes learned from the raw streams. To speak, it names the two objects whose word associations are strongest during the event, each with its best word, in the order in which those associations peak.

ArmPairs seen in trainingKnown words, new rolesOne new wordTwo words only ever named
Learned words and classes1.001.000.750.98
True object kinds1.001.001.001.00
The same words in random order0.500.500.390.50
Associations learned against another episode's scene0.270.270.050.00
Table 8: Share of test events in which the creature says "A hits B" with the right words in the right order, three seeds pooled, 68 test events per seed. Chance is 0.05.

In 194 of the 196 events where both words were right, the order was right too, including events with two words the creature had only heard as names. The weak point is word choice for a new word next to a known one (0.75, and 0.50 on one seed): all eight errors said the same wrong object word, and in each of those sentences the order was still right. Order comes from when each object's support peaks. Ordering by when an object first appears fails, because the patient is visible before it moves.

Two machines that rely on each other

The caretaker is replaced by a second creature of the same kind. Each creature has its own perception and its own word–object associations, so their internal object classes do not correspond: they share only words. Hunger alternates between them, and neither is told its role. The hungry one may say a word for what it wants. The full one hears it and may hand over the object the word means to it. Eating is not a choice the creature makes: a hungry creature eats any food in its view until its energy is full, so food handed to it is eaten. A shape that is not food is handed over but not eaten, and a full creature does not eat at all. The giver gains energy only when the receiver eats what it gave. The key test comes after the food has changed, when both face a first try at once: the hungry creature needs a food word it has heard but never said, and the full one hears a word it has never been asked with. With a self-check, a speaker says a word only if its own reading of that word returns the object it wants, and otherwise stays silent.

PairAsker says the never-said food wordGiver follows a never-asked wordGiver gives the food on a non-food wordFed, all requestsFed through the real word
Both grounded, with self-check0.830.890.000.750.75
Both grounded, without self-check0.830.810.000.670.67
Both trial and error0.000.470.930.940.00
Random0.130.150.160.110.02
Table 9: First tries at the start of the test, three seeds pooled, both creatures. "Follows" means the giver hands over the object the heard word names, food or not. Words are clean syllable tokens; objects are each creature's own learned classes. With the associations shuffled (20 sets per creature), the asker says the food word in 0.15 of first tries, and the giver's reading of the asker's word reaches the food in 0.21.
  • Words are the only shared code, and they carry it. The asker says a word it has never said, and the giver acts on a word nobody has asked it with. With shuffled associations both fall to chance.
  • Trial and error builds a private code. The trial-and-error pair is fed more often at the test (0.94), through conventions that differ per seed and per direction, and its giver answers any word with the food. It never uses the shared word.
  • Roles are learned where they pay. The full creature gives in nearly every episode from the first block on, and the hungry one asks in at least 0.97 by the end of the first phase. Habits that cost nothing, the full creature speaking and the hungry one also giving, are not unlearned.
  • The self-check protects listening. Without it, a hungry creature that could not yet say the new word said its nearest word instead. On the one seed where that word named the wrong object, nobody ate, and the giver learned that acting on words did not pay: at the test it still understood the new word (1.00) but gave on it only 0.55 of the time. With the self-check the giver kept listening (0.89).

Feeding each other: reciprocity and trust

Two more questions were asked of creatures that feed each other. Does giving that costs the giver last because the partner gives back later? And does a creature learn which partner it can trust? Both are questions about behavior, so true object kinds stand in for perception in this step.

Reciprocity. Energy now carries over between episodes and falls over time. A gift costs the giver 0.17 of energy, against about 0.8 that a hungry creature gains when it is fed, and the giver gains nothing when the receiver eats. A gift can pay only through what comes back later. In one condition each episode closes the learner's credit, in a second the credit reaches across episodes, and in a third the creature also remembers what its partner did the last time it was asked.

ConditionGives when askedMean energy
A creature living alone–0.25
Each episode closes the credit0.060.26
Credit across episodes0.050.25
Credit across episodes, with a memory of the partner0.280.34
Table 10: Two grounded creatures over the last 100 episodes, three seeds pooled, cost of a gift 0.17.
  • Reciprocity did not appear. With credit across episodes the pair gives as rarely as with each episode closed, and ends up as well off as a creature living alone.
  • Not giving is a stable state. While nobody gives, a gift is never followed by being fed, so nothing carries back that would make the gift worth its cost.
  • The memory of the partner raised giving, but not conditionally. A creature gave no more often after its partner had given than after it had not (difference −0.00), so the extra giving is not tit-for-tat.
  • Trial and error gives often but blindly. A trial-and-error pair gives in 0.21 to 0.79 of asks across conditions, but hands over what was asked in only 0.08 to 0.18.

Trust. Three creatures share the world: two honest ones and an exploiter that is paid when a harmful kind it hands over drains the receiver. The hungry creature chooses whom to ask. Trust is learned from the creature's own dialogue, with the same learning that binds words to objects: for each partner, the creature learns which object that partner hands over in response to each of its words. It does not ask a partner from whom it expects an object of negative learned value. If no partner is trusted, it stays silent. At the start of the test phase the exploiter and one honest creature swap behavior without notice.

How the hungry creature choosesAsks the exploiterHarmedFedSilentHarmful gifts from the partner that turned, first 30 asks
At random0.510.650.150.0010.7
By learned value0.110.330.310.0010.3
Trust learned from dialogue0.020.010.290.332.2
Trust, then learned value0.020.010.310.332.0
Table 11: The two honest creatures as askers, last 100 episodes before the swap, three seeds pooled. The last column counts harmful gifts after the swap.
  • Trust keeps the honest creatures away from the exploiter. They almost never ask it and are almost never harmed. With a random choice they are harmed in 0.65 of their hungry episodes.
  • Trust is revised quickly. After the swap, a creature with trust is harmed about twice by its formerly honest partner and then stops asking it. By learned value alone that takes about ten harmful gifts. With trust and value combined, it took exactly two on every seed, for both askers.
  • Trust also catches honest mistakes. A hungry creature that cannot yet say the new food's word uses a substitute word, and on some seeds an honest partner reads that word as the harmful kind. By learned value alone, 44 of the 66 harmful gifts came from honest partners in this way. With trust, the only harmful gifts left came from the exploiter, two in all.
  • The price of trust is silence, not hunger. A creature with trust stays silent in 0.33 of its hungry episodes and is fed as often as one choosing by value.
  • Words stay shared with honest partners. What a creature expects from an honest partner agrees with the narrator on every word it said to that partner at least five times.
Video 2: Trust among three creatures, seed 1 (76 s). Three ways of choosing whom to ask run side by side on the same world: at random, by learned value, and by trust learned from dialogue. The film shows the first 15 episodes, the first 10 after the food changes and 30 around the swap; cards mark the stretches in between, and the counters and the chart include every episode. The chart counts harmful gifts to the honest creatures, and the dashed line marks the swap. Every frame is drawn from the recorded run; the creatures use true object kinds.

A second verb

Up to here, words named objects, and the machines used them to ask for objects and to hand them over. The only word for an event was "hits". A language also needs words for kinds of events: a verb has to mean the same thing whichever objects take part. So we returned from the exchanges between machines to the narrated world and added a second verb.

A second verb, "follows", is added next to "hits", on five filled 2D shapes in a 32 × 28 field. In a "follows" scene the leader crosses and leaves the view, and three frames later the follower enters and takes the same path. A verb needs something to bind to, so the learner also learns events, one object class becoming active and then another, with the same learning that binds words to objects. A verb word binds to these learned events as a noun binds to a class. Perception includes all four additions; words are syllable tokens.

Measure"hits""follows"
Hearing the verb picks its kind of scene0.60–0.740.83–0.96
Same, with events tied to one pair of shapes0.500.50–0.51
Saying the right verb for a seen event, known shapes0.99–1.000.80–0.94
Saying the right verb when new shapes take part0.78–0.940.38–0.71
Narrated as it happens, order right (order keyed by the verb)0.88–0.990.88–1.00
Hearing the verb, verbs shuffled during training (control)0.40–0.540.55–0.68
Table 12: The second verb, three seeds pooled; ranges over test categories (seen pairs, new roles, one and two new words) and both narrations, except the order row, which is the narration in the order of events. 0.5 is chance; the margin of hearing over the shuffled control is 0.22–0.27 per verb and narration.

Each verb word comes to mean its kind of event, including for pairs of shapes never seen with that verb. This needs events that do not depend on which shapes take part: events tied to one pair stay at chance, because each verb is heard with several pairs. Keying word order on the heard verb also helps "hits" with known words in new roles (0.98–0.99, against 0.65–0.76 without the key). Saying "follows" drops when new shapes take part; the drop occurs only in events with the disc, which is split over about three learned classes, so that a switch between two disc classes looks like the hand-over of "hits".

"A follows B" names the follower first, although it appears second, so its word order runs against the order of events. Associations keyed by the verb carry this order for words heard in "follows" sentences (0.83 on seen pairs), but not for new words (0.00–0.12). Carrying a verb's order over to new words needs a role memory with a supplied position slot (0.68–0.71). Children's early grammar shows the same pattern: word-specific constructions come before general ones (Tomasello 2003).

Words without a narrator

So far a narrator taught every word. Here two machines start with eight syllables and no words, and have to agree on their own. The hungry one needs one of six kinds, a different one each time and known only to it, so a single word for "now" would succeed only one time in six. Both see the same scene. Every word heard, including one's own, is associated with what is in view, as in naming. When the speaker is fed, it repeats its word. True object kinds stand in for perception in this step.

VersionHungry machine gets what it needsOne shared word per kind
Learning pair, no narrator0.99from episode 140–170
Narrator-taught pair, for comparison0.99from the start
Trial-and-error pair0.52never (one dialect per direction)
Without hearing one's own words0.44never
Separate views of the scene0.19never
Random speakers0.14–
Three machines, the third overhears0.99yes
Three machines, no overhearing0.780.56 of kinds
Table 13: Inventing words, three seeds; success in the last 200 episodes before the first new kind.
  • A shared vocabulary emerges. Two machines with no narrator settle on one word per kind, and from then on the needed kind arrives in 0.99 of exchanges. Success passes 0.9 between episodes 100 and 150. No word changed afterwards, also not when three new kinds arrived. Five and seven kinds work as well.
  • Every run invents a different language. Across the three seeds, no kind gets the same word in any of 18 comparisons. Within a run the words are fully shared.
  • What it takes: looking at the same thing, hearing oneself speak, and noticing that it worked. With separate views it fails (0.19), and without self-hearing it stays at 0.44. The repeat on success raises early success (0.68 against 0.35 in the first 100 episodes).
  • Groups need overhearing. With three machines and no overhearing, a misread word confirms itself, because the listener hands over its own reading and sees it arrive.
  • Where agreement is a real test. With true kinds, a shared view and self-hearing, the two machines' word stores are identical by construction, so once a word exists the pair cannot disagree. Agreement is a real test in the arms where the stores can differ: the trial-and-error pair and the groups of three.

The language chain under noise

With the noise addition on every pixel, the steps from naming to the two machines were run again at two noise levels, on three seeds, against their committed predictions. In the words-from-sound step the ear was kept clean and without the noise addition, because it reads speech as noise; in the order and describing steps the ear had noise and the addition as well. On clean input with the noise addition switched on, every pixel step reproduced its committed results.

StepCleanσ = 0.01σ = 0.02
Naming in never-named lanes, token words, word memory0.960.960.96
Naming from learned associations, token words, 600 episodes1.001.000.95
Words from sound, then naming, 600 episodes0.98 / 0.960.96 / 0.920.93 / 0.91
Asking with a never-said word, token words0.941.000.91
Two machines, asker says the new word, token words0.830.90–0.960.94–0.96
Who did what, words from sound, one / two new words0.97 / 0.900.92 / 0.900.72 / 0.57
Saying what happened, one / two new words0.75 / 0.980.78 / 1.000.17 / 0.32
Table 14: The language steps under white pixel noise, three seeds pooled.

At σ = 0.01 every prediction keeps its committed verdict except one control in the order test with token words: with the sentence spoken after the event, two new words reached 0.76, where the control was predicted to stay at or below 0.65. At σ = 0.02 the headline results for seeing, naming, asking and the two machines hold, though three of their secondary predictions fail (Appendix B): two by margins of 0.012 and 0.002, and one in the two-machine step, where a grounded asker paired with a trial-and-error giver said the new word in 0.58 of first tries, predicted ≥ 0.7. Sentences with two words only ever heard as names can no longer be ordered reliably. A diagnostic on the order test with token words found the cause on the visual side: at σ = 0.02 the two kinds that are only ever named merge into one visual class on two of the three seeds, and with one class covering both objects, a sentence and its reversal score the same. The words themselves still find their object (0.89). This is the limit seen in Table 3, reached in this world by a different pair of kinds.

What the system is given

The system learns object classes, word forms, the associations between words and objects and their timing, word order from timing, which object restores energy, and, in the later steps, verb meanings and a shared vocabulary. It is given the following.

  • The world. A synthetic gray-level pixel field with a dark background, five kinds with clearly different brightness waveforms (five 2D shapes for the second verb, six abstract kinds plus three new ones for inventing words), and white, independent pixel noise where noise is used.
  • The sound. Syllables as clean synthetic waveforms on one sensor, with no real audio, in the words-from-sound, naming, order and describing steps. Elsewhere syllables are clean tokens: cut by a simple statistical segmenter in the asking and two-machine steps and in the token-word arms, spaced so that there is nothing to cut in the second-verb step, and said as whole two-syllable forms when inventing words.
  • Four additions to the perception layer, each tested with a control (Table 2), with their constants fixed in advance.
  • Constants. The time window within which words and objects are associated, and the readout thresholds.
  • For asking and describing: the hunger drive and the caretaker, whole-word speech actions, speaking from the word–object associations, and the rule for which words have not yet been said. For describing, "hits" as the only verb and its place in the sentence, exactly two object words, the production rule, and a record of the last event.
  • For the two machines: alternating hunger, the reward coupling between giver and receiver, the give action, reading a heard word as an object, and the self-check.
  • For reciprocity and trust: true object kinds; energy carried over between episodes, its drop, and exhaustion; the cost of giving; the memory of the partner's last act; and, for trust, three creatures of which only the hungry one speaks, partners told apart by index, the exploiter's payoff, the swap, and the rule from expectation to decision (do not ask a partner whose predicted gift has a negative value).
  • For the second verb: the form of the learned events, which word forms are verbs, and, for the role memory, the side of the verb a word is on.
  • For inventing words: true object kinds, a shared view of the scene, hearing one's own words, the syllable inventory, and the repeat on success.

Limitations

  • The world is small and synthetic. Five kinds, one or two verbs, three-word sentences, synthetic syllables and a dark background. The results show what the learner does, not that it handles real images or real speech. No real camera or microphone has been used.
  • Noise. Recognition holds up to σ = 0.02 of white pixel noise. Beyond that, kinds whose brightness patterns lie closer than the noise tolerance merge. The noise addition assumes a channel that is quiet most of the time; on the ear, which is not, it reads speech as noise, and handling noise on busy channels is open.
  • Variation. Each syllable has two pronunciations, and all objects of a kind look the same. Real speakers differ in accent and dialect, and real objects in lighting and viewpoint. Whether more varied exposure makes recognition more robust has not been tested.
  • One kind, several classes. Under stronger noise, under rotation and for one 2D shape, a kind splits over several classes, and words learned on one class miss the others.
  • New places. A kind is named in a never-visited place from its second pass there on, not its first.
  • Order told after the event is not understood (at chance), because words are bound only to what happens within a short window around them.
  • Verb-dependent order for new words needs a supplied position slot. The learned associations alone carry it word by word.
  • Reciprocity did not appear, and trust was tested with true object kinds and with partners told apart by index, not recognized.
  • Supplied structure remains, as listed in the previous section. Several steps use true object kinds or token words where the question is about behavior rather than perception; the tables and the previous section say which.
  • Internal results. No outside group has rerun them.

Summary

Given the raw pixel and sound streams of a small synthetic world, one learning layer found the objects and the words, bound the words to the objects without labels, and read who did what to whom from timing alone. Using what it learned while listening, a creature asked for food with a word it had only ever heard, said what happened in an event it watched, and relied on a second creature with which it shared only words. Associations learned from its own dialogue told it which partner to trust. Two creatures without a narrator settled on a vocabulary of their own. Giving at a cost did not become reciprocal. Of 136 pre-registered predictions, 114 held, 4 held in part and 18 failed.

With perception switched off, the same learning layer carries out the commands of gSCAN and ReaSCAN at 100 % on every split. Together, the two reports follow one learner from raw signals to words, and from words to action. Both worlds are small and synthetic; real images, real speech and larger vocabularies remain to be shown.

What comes next

  • Richer worlds to see. The shapes here are simple, and each crosses the field on its own path. Next are shapes with more structure, an object recognized as one thing when it appears in several forms, and then a camera and a microphone in place of the synthetic streams.
  • More language. More verbs, real English words, and more grammar: longer sentences, word order that the learner works out for new words without a supplied position slot, and sentences about events that are already over, which it cannot yet understand.
  • Acting in unknown worlds. In this report a caretaker or a partner hands over what the learner needs once it has the word. ARC-AGI-3 takes that help away: its games come without instructions, so a learner has to find the objects, what its own actions do and what the goal is by acting. We are now testing the CLS there.

Appendix A: What the system learns

Every result in this report comes from the learning layer of the CLS, which learns incrementally from the raw streams, without pretraining or gradient descent. It learns:

  • object classes from pixel brightness, and word forms from the sound stream;
  • which words go with which object classes, and when they are heard relative to what is seen;
  • events, one object class followed by another, for verb words to bind to.

For wanting and choosing, the asking steps add the CLS's decision learner, unchanged. What the learner has stored can be read back, so a statement such as "this word activates that object" can be checked directly.

Appendix B: Pre-registered predictions, graded

Every scored step had its protocol, predictions and refutation criteria committed to version control before it ran. None of the graded refutation criteria was met in the clean runs. Under noise at σ = 0.02, the describing step met its refutation clause for one new word (Table 14).

StepPredictionsHeldFailedWhat failed
Objects from pixels, none of the four additions (Table 2 control)64, and 1 in part1Kinds named in trained lanes 0.06 by the protocol's measure, predicted ≥ 0.6; the prediction on new places then held only numerically.
Objects from pixels, first addition without its similarity check (Table 2 control)64, and 1 in part1Purity 0.70 and 0.85 on two seeds, predicted ≥ 0.9; recognition held as measured but is inflated, because kinds share classes.
Objects from pixels880–
Naming learned classes64, and 1 in part1In part: the learner without the four additions named worse than the full learner on every seed, as predicted, but better than the predicted ceiling (0.63–0.70, predicted ≤ 0.5). Failed: the shuffled control strayed up to 0.12 from chance in four of six cells, outside the band of 0.1, with no sign of naming above chance.
Naming from learned associations, token words660–
Words from sound440–
Who did what to whom, words from sound651A comparison arm with token words and the simple segmenter was predicted to show a segmentation artifact on seen pairs (at least 0.1 below new roles). It reached 1.00 on both, so the run does not settle whether the learner's own segmentation or the spacing of words in this world avoids the artifact.
Asking871On one seed, the grounded creature learned the first food less reliably (0.28 in the last 50 episodes, predicted ≥ 0.6): it credited a kind that often crossed during meals.
Saying what happened761One new word next to a known one: 0.75, predicted ≥ 0.8.
Two machines13121With the true object kinds, the giver read a never-asked food word right in 0.92 of tests, predicted ≥ 0.95: one word form was never bound during listening on one seed, which capped the giver there at 0.75.
Reciprocity and trust104, and 1 in part5Reciprocity: the trial-and-error pair gave in 0.50 of asks with each episode closed, predicted ≤ 0.15; credit across episodes did not raise giving (0.05 against 0.06, predicted at least 0.25 more); the memory of the partner did not make giving conditional (difference −0.00, predicted ≥ 0.3); the gain from credit across episodes was predicted to be smaller at the higher cost, but it was +0.03 against −0.01, both about zero; with the partner memory, the asker said the new food word on its first try in 0.46 and 0.79, predicted ≥ 0.8. In part: after the swap, trust cut asks to the partner that turned to 0.20–0.30 within the first ten asks, predicted ≤ 0.2, and to 0.00–0.15 in the next ten.
New places12120–
Noise11110–
Second verb15105Hearing "hits" in new roles 0.60 in one of 16 cells (predicted ≥ 0.65); a shuffled-verb control above 0.6 in five cells; saying "hits" with two new words 0.78 (predicted ≥ 0.8); without the verb key, "A follows B" still came out in the right order in 0.27 of seen pairs (predicted ≤ 0.2); the keyed shuffled-verb control 0.59 (predicted ≤ 0.5).
Words without a narrator18171Without the repeat on success, agreement was predicted to come later on every seed. It came later on two seeds and earlier on the third (episode 120 against 170). One prediction that held, that guesses favor the newest kind, held as worded but could not be told apart from chance (0.16 against 0.13).
Table B1: Pre-registered predictions per step, clean runs: 136 predictions, 114 held, 4 held in part, 18 failed. An earlier order test with token words, which the order test with words from sound replaces, had 5 predictions, of which 3 failed on a segmentation artifact of the token segmenter.

In the replication under noise, the original predictions were graded unchanged at each level. At σ = 0.01 one verdict changed: in the token-word order test, the control with the sentence spoken after the event reached 0.76 on two new words, predicted ≤ 0.65. At σ = 0.02 these failed in addition to the clean failures: in naming from learned associations, one prediction by 0.012; in asking, one prediction by 0.002; in the two machines, one (the asker of a grounded asker and trial-and-error giver pair, 0.58 against 0.7); in the token-word order test, one (two new words); in the order test with words from sound, three (both new-word categories, 0.72 and 0.57, the timing-blind control at 0.61, and the true-kinds arm at 0.83); in describing, three more, and its refutation clause was met for one new word.

References

  • Tomasello, M. (2003). Constructing a Language: A Usage-Based Theory of Language Acquisition. Harvard University Press.

This report describes internal research results. They have not been independently replicated. It makes no claim about products or their performance.

Investment Opportunities

We are currently engaging with investors and strategic partners interested in long-term technological impact grounded in scientific discipline.

Wakeline represents a fundamentally different approach to artificial intelligence, prioritizing continuous adaptation, reduced compute dependence, and real industrial application.

Conversations focus on collaboration, evidence building, and shared ambition.