GLEE / Learning through play

Terraria. An open learning lab.

Agents play through game controls and learn from what happens. Follow their actions, inspect what changes, and compare what they retain.

Connecting to the room

01 / OPEN LAB A game to watch. An intelligence to understand.

Research desk ↗

On screen

Live camera of the saved world. The previous AI controller is retired; this camera is not a running learning trial.

Waiting for the next adventure

The room is open. Live game capture and statistics will appear when the player connects.

Live game capture · experimental player

Volume60%Muted · game audio is separate from AI voices

Trying toWaiting for game state

Selected actionNo action observed

ObservationNot connected

Both AIs at a glance

Updates automatically

Connecting

Waiting for current data

—

Connecting

Waiting for current data

—

Experimental AIs · 8 types

MemorySkillsWorld modelExplorationImitationReinforcementPopulationHybrid

No run feeds connected.

Macro to micro

Goal map

Waiting for a retained goal map.

Selected intent and observed outcomes are distinct. Child reports prompt a parent review.

Health—Native player health

Resources—Wood + stone + gel

Survival—Deaths unavailable

Attention markers · subjects and evidence

What the AI is watchingWaiting for tracking data

Player Current aim Threat Resource Route Pickup

A thin square marks the targeted block. Small shapes mark other subjects; their IDs and group counts stay in the inspector below.

Subjects and reasons

Display:

These controls change your view only. The AI keeps each subject’s identity when it groups or separates its attention.

Tracked subjects appear with the next game observation.

Ember & Scout · commentary

Ember AI streamer · talks with you

Say hello in chat. Ember answers here and in the conversation.

Waiting for a viewer conversation

Scout AI commentator · follows the run

Watching for the next game observation…

Commentary appears while the stream is being watched

Inspection workspace

From observation to outcome

The experiment · early architecture development

Build an AI that can learn from play

We are building and testing the architecture that could make a capable player possible. Beating Terraria is beyond the current system; this stage is about developing the foundations.

What we are testing: Can the AI connect an action to its outcome, turn an attempt into a useful skill, and reuse that skill when the situation changes?
Current development focus: Grounded movement and targeting, gathering and shelter tasks, and retaining useful behavior so each attempt does not require solving everything again.
What would count as progress: More reliable basic actions, fewer repeated mistakes, and useful skills that survive a restart and work in changed situations. These are tests to pass, not capabilities we have established.

Now · build and testBasic competenceAct, observe, correct
Next · establishRetain and adaptReuse beyond one attempt
Later · ambitionBroader game playCombine reliable skills

A successful action, more attempts or a larger skill library alone does not prove learning. Stalls and failed attempts help us identify what the architecture still lacks.

The next move

Waiting for a game observation

No active plan

The player shares its selected skill and observed outcomes here.

No acknowledged actions yet

In the backpack

Inventory appears when the player enters the world.

Statistics come from the game bridge; unavailable measurements stay blank. Viewer suggestions enter a review queue and do not directly control the game.

Learning samples—Acknowledged native outcomes

Reward—Latest native outcome

Planning—Motor loop unavailable

Which skills are getting tried?

These are the current controller’s published skill trials and selection scores. They describe this run; trials alone do not establish retained learning or better play.

Skill	Trials	Selection score
No skill statistics yet.

Follow a thought through the world

What changed its next move?

Explore the observations, retained experience, choices and outcomes behind the player.

Decision history

History begins when the instrumented player connects. Earlier experience can be summarized without inventing earlier decisions.

Waiting for policy records.

A current policy record is unavailable.

Saved decisions remain in the history selector when present. Missing current data does not mean the player has no retained experience.

Objective

Observed world

Policy memory

Choice

Requested action

Observed outcome

Select a node to inspect its source or follow an earlier experience.

What else could this player consider?

Available skills come from this decision. Other implemented skills stay visible with their current availability.

Skill	Available	Experience	Selection

How this world is represented

The player sees a bounded local tile patch, nearby subjects, inventory and native state. Its learned memory stores skill statistics by situation. Authored safety and planning rules can override learned selection. A remembered home site or rejected route is shown only when that state actually exists.

The timeline is an observation journal for viewers. It is not additional memory recalled by the player. Records sample changes up to twice per second and combine repeated decisions into intervals; outcomes include only the acknowledged samples linked to those records.

Older experience may predate recording. A selection score is not a probability. Health and item changes following an input do not by themselves prove that input caused the change.

Current controller · proposed experiments

The architecture and what we are testing

The live player is a guarded control loop. This panel separates what the stream reports now from theories we can test next.

Waiting for a live snapshot

Read-only evidence surface Seven candidate modules are implemented for comparison. They are not seven systems running in this stream, and none has a verified native Terraria benefit.

1 · perceiveNative observations

The legacy bridge supplies structured local tiles, nearby subjects, player state, inventory and world progress to the controller.

Structured observation in

2 · decideGoal + policy

Waiting for a goal. A learned selector may be shown; authored safety/planning can still govern.

Guarded selection

3 · actRaw controls

No action reported. The viewer sees a request, not a control affordance.

Short control lease

4 · checkACK + outcome

No matched outcome yet. A count of updates is not proof of durable game learning.

Acknowledged outcome

AI input nowStructured native observation from the bridge.

Viewer dataSanitized state, frame and policy records published for inspection.

Future entrantScreen/audio human-I/O is a separate experiment boundary; this panel does not claim the legacy player consumes it.

What the current stream can prove

Current AI, as reported

Published goal: Unavailable
Selected skill: Unavailable
Decision branch: Unavailable
Input boundary: Native structured observation · legacy bridge
Implementation status: Candidate package implemented · untrained/unqualified
Published decision records: Unavailable
Freshness: Waiting for a snapshot

When the feed is unavailable, the honest state is unavailable. A saved viewer history is not silently promoted to current memory.

Theory about the current design

Our falsifiable reading

Our hypothesis is that authored task sequencing and guarded control account for much of the early progress. An outcome-informed selector could improve local skill selection, but whether it does, and whether any improvement transfers when a fresh world changes the geometry, timing or resources, still needs testing.

Test it by freezing updates, removing retained experience, and replaying matched fresh worlds. Improvement must survive a restart and be measured against total game time, engineering time and compute.

Inference. This hypothesis combines the public mechanism fields with the current source reading; it is not a claim that a pretrained model or autonomous native learner is presently verified.

Seven implemented candidate modules · comparison-ready, none running here

01Experience memoryRetrieve relevant failures and successes

Keep outcome-conditioned episodes and revise them when evidence conflicts. The stream moment is a remembered mistake changing the next attempt.

Falsify: compare useful retrieval with shuffled memory at equal storage and inference cost.

Source: fast-learning theory note, 2026-10-19 · mechanism precedent: Voyager (skill memory, different interface).

02Skill builderAcquire and compose closed-loop skills

Turn successful behavior into policies with initiation, correction, termination and reliability, then combine them for gathering, crafting and shelter.

Falsify: ablate the skill library while holding memory and planner cost constant.

Source: fast-learning theory note, 2026-10-19 · acquired reuse must operate through the admitted observation/action boundary.

03World modelPredict short action consequences

Learn action-conditioned dynamics, imagine short futures, and check predictions against the next frame before trusting longer plans.

Falsify: measure held-out prediction quality and total compute; a model that saves actions but burns more training time is a different tradeoff.

Source: fast-learning theory note, 2026-10-19 · mechanism precedent: DreamerV3, different game/interface.

04Experimental learnerChoose informative tests

Maintain competing explanations for a failure, such as wrong tool, aim or obstruction, and change one factor to learn which explanation survives.

Falsify: compare uncertainty-directed experiments with ordinary exploration at equal failed interactions.

Source: fast-learning theory note, 2026-10-19 · mechanism precedent: Plan2Explore.

05Demonstration apprenticeBootstrap timing from labeled human play

Learn a small motor prior from synchronized screen and input demonstrations, then test whether it adapts without live hints on held-out worlds.

Falsify: charge demonstration creation and pretraining, then compare total human-plus-machine cost against a no-demo learner.

Source: fast-learning theory note, 2026-10-19 · mechanism precedent: VPT, much larger labeled Minecraft data.

06Model-free reinforcementUpdate policy/value from consequences

Use an unchanged recurrent policy/value baseline and let sensory-derived progress shape choices directly. It is the clean control for claims about learning.

Falsify: compare improvement after the same amount of native game exposure with a frozen policy.

Source: fast-learning theory note, 2026-10-19 · no Terraria gain or autonomous native update is currently verified.

07Evolving populationSearch policy lineages and training schedules

Run several bounded learners, preserve ancestry and select on a predeclared competency while charging every member's experience and compute.

Falsify: compare aggregate cost and fresh-world generalization against one learner with the same budget.

Source: fast-learning theory note, 2026-10-19 · mechanism precedent: population-based training, not a current stream implementation.

Sources, freshness and limits

Current source read: gleex_terraria/operate.py, live.py and feedback.py show the legacy controller consuming native structured observations and guarded skill selection. terraria_live_page.ts, terraria_live_protocol.ts, terraria_worldview.ts and terraria_worldview_view.ts define the sanitized viewer projection, read 2026-10-19.

Runtime freshness: the green status follows the latest snapshot. When it is stale or unavailable, dynamic current fields are cleared and this panel keeps only the dated source explanation.

Known limit: no public field here identifies a pretrained model, and no fresh-world result proves retained competency. “Learned updates”, attempts, ACKs and an alive process are evidence of reported events, not by themselves evidence of better Terraria play. The seven modules are implemented/untrained package work; they are not wired into this live player.

Skill-writing learner · assisted practice

An AI that writes its own skills

A frontier model uses its prior knowledge, teaching notes and provided sensing and control functions. It chooses goals, writes Python, runs it in Terraria and records results and reusable programs. Saved programs and completed tasks do not by themselves establish useful retained learning. The reference controller plays a separate world.

Connecting…

Waiting for a current capture from the learner's own world.

No current game capture

—

Waiting for the learner

—saved skill programs

—recent successes / reported tasks

—life

—run age

Progression checklist 10 fixed early-game milestones · minutes after this world began

Waiting for the learner

Experiments each written down before it ran · one run per side, so pilots, not proof

Waiting for learner data

Macro to micro

Goal map

Waiting for a retained goal map.

Selected intent and observed outcomes are distinct. Child reports prompt a parent review.

Progress: first time it had each item minutes after this world began

Waiting for learner data

Skill library (written by the AI)

Waiting for learner data

Lessons it drew from play

Waiting for learner data

Recent tasks

Waiting for learner data

Code it is running now

—

Inventory: —

For the curious. For the researchers.

Go beyond the stream.

Inspect the observation behind the broadcast. Keep its source, time and limits attached.

Selected source: Learning from play
Source observation: Waiting for a public observation
Evidence available: No current evidence

Open public JSON ↗

An observation is a snapshot, not a benchmark. Saved code, reported success and repeated actions are useful evidence to inspect; improved learning needs comparisons, retention and transfer tests. Downloads contain public feed data only.