OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

TL;DR

OmniLottie is an autoregressive multimodal framework that generates Lottie vector animations from text, text+image, or video instructions. The central trick is a Lottie tokenizer that flattens the format’s nested JSON into compact command/parameter sequences with an offset-based discrete vocabulary, then trains Qwen2.5-VL on those tokens — so the VLM never has to reproduce JSON metadata. A 2M-sample dataset (MMLottie-2M) is built from web-crawled Lottie files plus a motion-library augmentation that animates static OmniSVG assets with templates extracted from native Lottie files. On the introduced MMLottie-Bench, OmniLottie reaches 88.3% / 93.3% / 88.1% success on the three tasks versus 12.7% / 28.0% / 9.2% for GPT-5, and beats the commercial Recraft tool on FVD and motion alignment.

Key claims

Raw Lottie JSON is a bad target representation: formatting tokens dominate the sequence and even DeepSeekV3 / GPT-5 produce valid Lottie at <13% success rate when prompted to emit JSON directly [Table 1, §1].
A custom Lottie tokenizer that splits the format into core properties, conditional properties, and five layer-type-specific Python objects, with offset-based discrete IDs per parameter category, raises Qwen2.5-VL’s success rate from 0% (raw JSON fine-tune produces 13.4%) to 97.3% / 92.0% / 90.7% on Text-/Text-Image-/Video-to-Lottie respectively [Table 3, §5.4].
Co-training on a mix of native Lottie (0.7) and SVG-derived animated Lottie (0.3) is optimal: pure Lottie undertrains geometry, pure SVG biases motion to be too simple, and 0.5+0.5 mixes lose motion alignment [Table 2, §5.4].
Motion-library augmentation (extracting transform trajectories from 1M native Lottie files and re-applying them as templates to static SVG-derived Lottie) is what closes the path-distribution gap between converted-SVG and authentic Lottie content [§3.1].
On the real-subset MMLottie-Bench, OmniLottie achieves FVD 202.14 / 180.27 / 227.11 vs. GPT-5’s 715.73 / 546.65 / 639.13 and Recraft’s 300.70 (T2L) / N/A / N/A across the three tasks [Table 1].
Against the optimization-based AniClipart and Livesketch baselines for Text-Image-to-Lottie, OmniLottie runs in 88.6s (vs. 1212s and 723s) at higher success rate and lower FVD [Table 1, §5.3].
The token sequence is substantially richer than baselines (21k–41k tokens vs. ~1k for raw-JSON LLM emissions), reflecting that the tokenizer pushes the model toward emitting actual animation content rather than boilerplate [Table 1, ”# Tokens” column].
Failures remain: autoregressive decoding still produces invalid sequences ~10% of the time, and complex animations exceed available context length [§6, Limitations].

Method

The system has two pieces: a Lottie tokenizer and a fine-tuned Qwen2.5-VL backbone.

The tokenizer reorganizes Lottie’s hierarchical JSON into a small set of core properties (v, fr, ip, op, w, h, nm, ddd, layers) plus conditional ones (assets, markers, fonts, chars) that are emitted only when needed. Each layer is parsed by type — Precomp (0), Solid (1), Null (3), Shape (4), Text (5) — and serialized into a flat sequence of commands and parameters. Continuous Lottie parameters are mapped to discrete tokens via offset-based encoding: each parameter type (temporal / spatial / transform / style) gets a disjoint vocabulary range with a type-specific scale factor, so token IDs never collide and each ID category carries a clear semantic role.

The backbone is Qwen2.5-VL with the Lottie vocabulary added as a new set of randomly initialized embeddings. Training is plain cross-entropy next-token prediction conditioned on the multimodal instruction (text / text+image / video). Decoding produces a Lottie token stream that the tokenizer detokenizes back to a valid JSON.

The MMLottie-2M dataset is constructed by (i) crawling LottieFiles, IconScout, Flaticon, Iconfont, and Icons8 to get native Lottie files, (ii) generating an auxiliary set by applying templated motions (translation / zoom / rotation) to static OmniSVG assets, (iii) spatial+temporal normalization to canvas and a 0–16 timestep range, and (iv) a motion-transfer pipeline that mines transform trajectories from 1M native Lottie files into canonical motion templates, clusters them by caption keywords, and injects them into the SVG-derived animations. VLM-driven annotation produces coarse-then-fine captions, single-frame conditioning prompts for Text-Image-to-Lottie, and rendered videos for Video-to-Lottie.

Results

MMLottie-Bench has a Real Subset (450 artist-designed samples, 150 per task) and a Synthetic Subset (GPT-4o prompts, Gemini3.1-Pro vector-style images, Seedance 1.0 reference videos, 150 per task).

Real-Subset highlights (success rate / FVD / Motion-Alignment from [Table 1]):

Task	Method	Success	FVD ↓	Motion ↑
T2L	GPT-5	12.7%	715.7	0.71
T2L	Recraft	77.3%	300.7	4.68
T2L	OmniLottie	88.3%	202.1	5.94
TI2L	GPT-5	28.0%	546.6	0.95
TI2L	AniClipart	87.3%	266.5	3.47
TI2L	Livesketch	91.3%	868.2	2.42
TI2L	OmniLottie	93.3%	180.3	4.44
V2L	GPT-5	9.2%	639.1	—
V2L	Gemini3.1-Pro	0%	1076.2	—
V2L	OmniLottie	88.1%	227.1	—

Synthetic-Subset numbers are directionally consistent — OmniLottie maintains 80–93% success while all general-purpose LLMs stay under 23% [Table 1].

Ablation evidence for the tokenizer is sharp: a Qwen2.5-VL fine-tuned on raw JSON tops out at 13.4% / 15.9% / 10.2% success on T2L/TI2L/V2L; swapping in the parameterized tokenizer (same backbone, same data) takes those to 97.3% / 92.0% / 90.7% [Table 3]. Data-mix ablation favors 0.7 native Lottie + 0.3 SVG-derived; pure Lottie drops Object-Alignment to ~1.85, pure SVG drops Motion-Alignment to ~2.35 [Table 2].

Latency: OmniLottie is 33–110s per sample versus 723s (Livesketch) and 1212s (AniClipart) for optimization-based vector-animation methods; against API LLMs the comparison is roughly even (16–60s) but the API methods rarely produce valid outputs.

Why it’s interesting

OmniLottie completes a three-paper triangle on the wiki for Vector Graphics Generation: VecGlypher: Unified Vector Glyph Generation with Language Models emits raw SVG <path> strings autoregressively for glyphs, Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure keeps SVG paths intact and orchestrates an LLM/VLM to emit a CSS/HTML animation layer, and OmniLottie picks the third design — keep the autoregressive-LLM-emits-program approach but design a custom tokenization layer so the program text stops looking like JSON. The three together bracket the design axes for vector-graphics-LLMs: (a) what is the model emitting (paths vs. animation overlays vs. parameterized commands), (b) is the tokenization stock subword or custom-discrete, and (c) is the geometry fully model-controlled or partially preserved from input. OmniLottie’s headline ablation — 13.4% → 97.3% success from swapping tokenization alone on the same VLM and data — is a clean piece of evidence for the The flavor of the bitter lesson for computer vision thesis that representation choice is the load-bearing decision even when “just scale the LLM” is on the table. It also pairs with VecGlypher’s finding that flagship LLMs collapse on glyph SVG — both papers argue that vector-graphics programs are underrepresented in standard LLM training corpora and that targeted data + targeted tokenization closes the gap that scale alone does not.