Sentdex — general multimodal LLMs may sidestep VLA/WAM research for high-level robotics intelligence

TL;DR

Sentdex (Harrison Kinsley) posts a self-contained thesis with a working demo: general-purpose multimodal LLMs — specifically GLM 5.3 Flash, and reportedly DeepSeek V4 Flash + Qwen 3.8 27B as well — can serve as the high-level robotics intelligence layer that VLAs, WAMs, and video-action models are being purpose-built for, with the LLM invoking the robot’s own high-level SDK (movement, arm joints, gripper open/close) as tool calls over camera frames. The demo runs on an XGO mini quadruped-with-arm, is fully zero-shot with no fine-tuning or robotics-specific data, and completes his years-long personal target task of picking up toys off the floor. He argues the hard part of robotics is intelligence / logic / planning under real-world imperfection — not object detection or gait — and that the VLA training pipeline (simulator, sim2real, teleop data collection, per-task brittleness) may be work that a fast, cheap multimodal LLM makes unnecessary for many manipulation regimes. Locomotion is called out as the one part he doesn’t yet think this covers, though he suspects it’s close for quadrupeds.

Key claims

GLM 5.3 Flash, invoked as a controller over the robot’s high-level SDK with per-frame camera input, drives an XGO mini quadruped-with-arm to pick up toys off the ground zero-shot, with no fine-tuning and no VLA/WAM/robotics training [tweet body].
The same pattern reportedly works with DeepSeek V4 Flash + Qwen 3.8 27B (vision model); an experimental vision version of DSV4F is anticipated to be the fastest of all when it lands [tweet body].
The robot / arm used is a Kickstarter-era demo build; the specific hardware “doesn’t really exist in the wild yet,” so the model is not benefiting from memorized robot-specific data [tweet body].
Framing claim: the hard part of robotics is not object detection (long solved) but “intelligence, logic, and planning for all the ways the real world just simply isn’t perfect” — the layer general LLMs are best at [tweet body].
Framing claim about VLAs: they are “super finicky” and always run into sim2real issues especially around the camera; teleop overcomes sim2real but produces per-task per-robot data debt; egocentric general-purpose VLA training may be sidesteppable entirely for this class of task [tweet body].
Locomotion is explicitly not claimed to be solved by this pattern, though quadrupeds are flagged as plausibly near (no fast IMU loop required for balance if gait is offloaded) [tweet body].
Personal-milestone claim: this is the first generalist solution he has gotten to work for the “pick up toys off the ground” task he has been trying to solve since becoming a dad of twins [tweet body].

Method

The tweet is the primary artifact — an accompanying video is attached but no code or write-up is linked. The described stack, reconstructed from the tweet body:

Robot: XGO mini — a small wheeled/legged quadruped platform with an attached arm and gripper, exposing a high-level SDK (per Sentdex: movement, arm-joint targets, gripper open/close).
Perception: raw camera frames from the robot’s onboard camera, fed to the multimodal LLM.
Policy: the multimodal LLM (GLM 5.3 Flash in the demonstrated run; alternates mentioned: DeepSeek V4 Flash + Qwen 3.8 27B, with the vision variant of DSV4F flagged as the anticipated fastest option) reads the camera frame, decides on the next SDK call, and iterates.
No training: no VLA, no WAM, no fine-tuning; the LLM has never seen this robot’s data at meaningful scale (it’s a Kickstarter demo build) and is invoked in-context only.
Speed axis: Sentdex emphasizes he wants the “logic” model to be as fast as possible while still intelligent — the framing motivates picking Flash-tier open-weights models over larger reasoning models.

The failure mode he explicitly rules out from personal experience is the VLA path: sim training pipelines, teleop-data-collection labor, per-task fragility to camera / lighting / setup, and the fact that “many edge cases” break VLA demos even under favorable conditions.

Results

Only qualitative results are reported:

A working video of the XGO mini picking up toys off the ground under GLM 5.3 Flash control, zero-shot.
Verbal confirmation that DSV4F + vision-capable Qwen 3.8 27B also works well on the same setup; Qwen 3.8 27B alone is asserted to “surely” work but not confirmed on tape.
No per-task success rate, N, latency numbers, comparison against a trained VLA, or hardware-cost breakdown. No held-out task set. No code or repo.

Why it’s interesting

This is the sharpest indie voice on the “LLM-as-controller sidesteps the VLA stack” thesis this quarter and adds a specific configuration to the pattern the wiki already tracks. On VLA Models the debate is dense: How Claude Performs on Robotics Tasks showed every LLM-guided MolmoAct configuration on LIBERO underperformed MolmoAct alone (though newer Opus generations defer less on novel tasks); Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) and Gemini 3.7 Flash saturates Robocurve physical tool-use benchmark at 92% vs 32% for 3.6 Flash (three weeks apart) posted zero-shot Claude Opus 5 and Gemini 3.7 Flash driving physical arms through the Inspect Robots harness (Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve)); LeRobot demos Claude Code operating SO-ARM101 zero-shot via Anthropic MHS — self-calibration to 4.1mm accuracy demonstrated Claude Code operating a SO-ARM101 through Anthropic’s Model Hardware Standard (Previewing the Model Hardware Standard) with no VLA in the loop at all. Sentdex adds a distinctive fourth cell to that grid: (a) open-weights frontier LLMs (GLM 5.3 Flash / DSV4F / Qwen 3.8) rather than closed frontier models, (b) quadruped + arm rather than the tabletop arms in the Robocurve/LeRobot lines, and (c) no dedicated eval harness — the robot’s own vendor SDK is the tool interface. This makes it the closest instance on the wiki to the “why not use general multimodal LLMs?” question posed against the Malik sensorimotor counter-position (Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level). It’s also a working data point for the “explore then compile” pattern MHS names formally: the LLM discovers what the SDK primitives can do at runtime, no upfront tag layer required. The gaps are the ones the tweet flags itself — no numbers, no locomotion story, no failure taxonomy — but as a demo-and-thesis pair it’s a load-bearing piece of evidence for the recipe-lever board’s “no explicit policy training needed once the base model clears a capability threshold” claim, sibling in spirit to Lerrel Pinto — in-context learning for robots is not that hard and to the emergent-ICL findings of GEN-1.5: Embodied Foundation Models are One-Shot Learners from the opposite (trained-VLA) direction.