FigmaTrace: A Comprehensive Training Dataset for Figma Design Workflows

TL;DR

FigmaTrace (Patronus AI) is an open-sourced dataset of 3,469 real Figma design trajectories capturing 200+ hours of expert designers doing complex long-horizon design work across 10 skills (accessibility fixes, templating, prototype wiring, sketch-to-Figma, platform adaptation, etc.). Trajectories are converted from screen recordings into agentic training data via design-phase-based segmentation rather than fixed-length context sharding. Fine-tuning open VLMs on FigmaTrace produces cross-domain generalization: a Qwen3.8-27B fine-tune reportedly matches or beats Claude Opus 5 and GPT-5.6-Sol on out-of-domain computer-use benchmarks (GUI-Odyssey, AndroidControl), with the eval-side gains attributed to element-selection accuracy, decisiveness (grid-overflow errors eliminated), and coordinate-emission behavior.

Key claims

The dataset contains 3,469 Figma trajectories over 10 design skills (2,883 train / 586 eval), covering both closed-ended tasks (accessibility, templating fixes) and open-ended ones (prototype wiring, sketch-to-Figma, platform adaptation) [§Dataset].
Design-phase-based trajectory curation beats maximum-context-length sharding by 7.3 absolute points on average (63.6 vs 56.3), because segmenting by design phase teaches skill-based grounding while fixed-length cuts break trajectories at points where task intent is unclear [§Results, RQ2].
Fine-tuning Qwen3.8-27B on FigmaTrace outperforms Claude Opus 5 on GUI-Odyssey (+6.4 abs) and AndroidControl (+11.8 abs), with gains up to +46 abs on AndroidControl over the corresponding baselines and +7.4 abs on the in-domain ScreenSpot-Pro Creative split [§Results, RQ1].
Manual inspection traces the qualitative gains to three patterns: element-selection accuracy (base median error ~457 px reduced to within ~15 px), decisiveness (grid-overflow errors on 10/150 GUI-Odyssey items eliminated entirely), and coordinate understanding (fine-tuned model always emits coordinates where the base often emits none) [§Results, RQ3].
The improvements transfer across model families: gains are reported on Introducing Muse Spark 1.1 Meta Glimmer, Gemma-4, and Qwen families, not only on the primary Qwen3.8-27B run [§Results].

Method

FigmaTrace is captured by recording professional designers executing 10 design-skill tasks in Figma. The raw signal is screen recording; the pipeline’s load-bearing step is converting recordings into agentic trajectories via design-phase segmentation — cutting on semantic phase boundaries (e.g. layout → styling → interaction wiring) rather than a fixed context-token budget. The dataset ships train/eval splits (2,883 / 586) and is released on Hugging Face (PatronusAI/figmatrace) alongside the best-performing Qwen3.8-27B fine-tune checkpoint.

Evaluation covers in-domain (ScreenSpot-Pro Creative split) and out-of-domain computer-use benchmarks (GUI-Odyssey, AndroidControl). Three research questions are ablated: whether human-captured workflows teach agentic navigation (RQ1), whether phase-based curation beats context-length sharding (RQ2), and which qualitative patterns in the data drive the gains (RQ3).

Results

Qwen3.8-27B + FigmaTrace beats Claude Opus 5 by 6.4 abs on GUI-Odyssey and 11.8 abs on AndroidControl; up to +46 abs on AndroidControl over baselines; +7.4 abs on in-domain ScreenSpot-Pro Creative [§Results].
Phase-based curation: 63.6 avg vs 56.3 for max-context-length sharding (+7.3 abs) [§Results, RQ2].
Qualitative: median element-selection error ~457 px → ~15 px; grid-overflow errors on 10/150 GUI-Odyssey items eliminated [§Results, RQ3].

Why it’s interesting

FigmaTrace is the first filed computer-use dataset targeting design as opposed to generic GUI navigation, and it operationalizes an intent-segmented alternative to the pretraining recipes on this wiki that mine screen recordings at fixed context lengths — VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos (masked-text SFT over unlabeled video) and Scaling Video Pretraining with Imagination Models (Photon-1) (Photon-1’s label-free next-latent-token pretraining on 18 years of screen recordings). It sits in the same design-space slot as Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (48k professional-software screen recordings) but ships expert design trajectories at 200 hours with explicit phase segmentation. The cross-family generalization claim (Qwen3.8, Gemma-4, Meta Glimmer all improve) makes this a dataset-side rather than model-side story.