What is $CODEC Robotics, Operators, Gaming? All of the above and more. Codec’s vision-language-action (VLA) is a framework agnostic model, allowing for dozens of use cases due to its unique ability to visualize errors in comparison to LLM’s. Over the past 12 months, we've seen that LLMs function primarily as looping mechanisms, driven by predefined data and response patterns. Because they’re built on speech and text, LLMs have a limited ability to evolve beyond the window of linguistic context they’re trained on. They can’t interpret sensory input, like facial expressions or real time emotional cues, as their reasoning is bound to language, not perception. Most agents today combine transformer based LLMs with visual encoders. They “see” the interface through screenshots, interpret what's on screen, and generate sequences of actions, clicks, keystrokes, scrolls to follow instructions and complete tasks. This is why AI hasn’t replaced large categories of jobs yet: LLMs see screenshots, not pixels. They don’t understand the dynamic visual semantics of the environment, only what’s readable through static frames. Their typical workflow is repetitive: capture a screenshot, reason about the next action, execute it, then capture another frame and repeat. This perceive-think loop continues until the task is completed or the agent fails. To truly generalize, AI must perceive its environment, reason about its state, and act appropriately to achieve goals, not just interpret snapshots. We already have macros, RPA bots, and automation scripts, but they’re weak and unstable. A slight pixel shift or layout change breaks the flow and requires manual patching. They can’t adapt when something changes in the workflow. That’s the bottleneck. Vision-Language-Action (VLA) Codec’s VLA agents run on an intuitive but powerful loop: perceive, think, act. Instead of just spitting out text like most LLMs, these agents see its environment, decide what to do and then execute. It’s all packaged into one unified pipeline, which you can visual into three core layers: Vision The agent first perceives its environment through vision. For a desktop Operator agent, this means capturing a screenshot or visual input of the current state (e.g. an app window or text box). The VLA model’s vision component interprets this input, reading on screen text and recognizing interface elements or objects. Aka the eyes of the agent. Language Then comes the thinking. Given the visual context (and any instructions or goals), the model analyzes what action is required. Essentially, the AI “thinks” about the appropriate response much like a person would. The VLA architecture merges vision and language internally, so the agent can, for instance, understand that a pop up dialog is asking a yes/no question. It will then decide on the correct action (e.g. click “OK”) based on the goal or prompt. Serving as the agent’s brain, mapping perceived inputs to an action. Action Finally, the agent acts by outputting a control command to the environment. Instead of text, the VLA model generates an action (such as a mouse click, keystroke, or API call) that directly interacts with the system. In the dialog example, the agent would execute the click on the “OK” button. This closes the loop: after acting, the agent can visually check the result and continue the perceive–think–act cycle. Actions are the key separator which turns them from chat boxes to actual operators. Use Cases As I mentioned, due to the architecture, Codec is narrative agnostic. Just as LLM aren't confined by what textual outputs they can produce, VLA’s aren’t confined by what tasks they can complete. Robotics Instead of relying on old scripts or imperfect automation, VLA agents take in visual input (camera feed or sensors), pass it through a language model for planning, then output actual control commands to move or interact with the world. Basically the robot sees what’s in front of it, processes instructions like “move the Pepsi can next to the orange,” figures out where everything is, how to move without knocking anything over, and does it with no hardcoding required. This is the same class of system as Google’s RT-2 or PaLM-E. Big models that merge vision and language to create real world actions. CogAct’s VLA work is a good example, robot scans a cluttered table, gets a natural prompt, and runs a full loop: object ID, path planning, motion execution. Operators In the desktop and web environment, VLA agents basically function like digital workers. They “see” the screen through a screenshot or live feed, run that through a reasoning layer built on a language model to understand both the UI and the task prompt, then execute the actions with real mouse and keyboard control, like a human would. This full loop, perceive, think, act runs continuously. So the agent isn’t just reacting once, it’s actively navigating the interface, handling multiple step flows without needing any hard coded scripts. The architecture is a mix of OCR style vision to read text/buttons/icons, semantic reasoning to decide what to do, and a control layer that can click, scroll, type, etc. Where this becomes really interesting is in error handling. These agents can reflect after actions and replan if something doesn’t go as expected. Unlike RPA scripts that break if a UI changes slightly, like a button shifting position or a label being renamed, a VLA agent can adapt to the new layout using visual cues and language understanding. Makes it far more resilient for real world automation where interfaces constantly change. Something I’ve personally struggled with when coding my own research bots through tools like playwright. Gaming Gaming is one of the clearest use cases where VLA agents can shine, think of them less like bots and more like immersive AI players. The whole flow is the same, the agent sees the game screen (frames, menus, text prompts), reasons about what it’s supposed to do, then plays using mouse, keyboard, or controller inputs. It’s not focused on brute force, this is AI learning how to game like a human would. Perception + thinking + control, all tied together. DeepMind’s SIMA project ihas unlocked this by combining a vision-language model with a predictive layer and dropped it into games like No Man’s Sky and Minecraft. From just watching the screen and following instructions, the agent could complete abstract tasks like “build a campfire” by chaining together the right steps, gather wood, find matches, and use inventory. And it wasn’t limited to just one game either. It transferred that knowledge between different environments. VLA gaming agents aren’t locked into one rule set. The same agent can adapt to completely different mechanics, just from vision and language grounding. And because it’s built on LLM infrastructure, it can explain what it’s doing, follow natural-language instructions mid game, or collab with players in real time. We aren’t far from having AI teammates which adapt to your play style and personalizations, all thanks to Codec.