Vector Graphics Generation

TL;DR

Generating editable vector graphics — SVG paths, CSS/HTML animation overlays, or Lottie JSON — by autoregressive language or multimodal models, rather than by rasterized image/video diffusion. The recurring lesson across three CVPR/arxiv papers from this wiki is that raw program text (SVG <path>, Lottie JSON) is not a free-lunch target representation: flagship LLMs collapse on it without large-scale targeted training data, custom tokenization, or task decomposition that preserves structure from input.

Key claims

• General-purpose flagship LLMs (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro) cannot reliably emit valid vector-graphics programs from prompt alone — they reach <13% success on Lottie generation (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens Table 1) and only R-ACC 46.65 / FID 19.59 on font-glyph SVG synthesis (VecGlypher: Unified Vector Glyph Generation with Language Models Table 8). The underlying claim both papers make: vector-graphics programs are underrepresented in standard LLM training corpora.
• Targeted fine-tuning on curated vector-graphics datasets closes the gap: a fine-tuned 27B Gemma3 hits R-ACC 100.5 / FID 3.46 on glyph SVG (VecGlypher: Unified Vector Glyph Generation with Language Models Table 8) and OmniLottie hits 97.3% generation success on Text-to-Lottie (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens Table 3).
• Representation/tokenization choice dominates over pure scaling within this domain. OmniLottie’s headline ablation isolates this: same Qwen2.5-VL backbone, same data, raw-JSON training → 13.4% success vs. parameterized-token training → 97.3% success on Text-to-Lottie (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens Table 3). VecGlypher reaches the opposite finding for its domain — subword tokenization of the raw d="" string suffices for fonts, and specialized SVG tokenizers add complexity without measurable gains (VecGlypher: Unified Vector Glyph Generation with Language Models §3.4). Conflict: these two findings disagree on whether custom tokenization helps; likely reconciled by format complexity (Lottie’s nested JSON has much more boilerplate than SVG path strings, so custom tokenization buys more), but neither paper tests the other’s hypothesis directly.
• Three distinct design axes have been explored on the wiki for what the LLM emits (covered by VecGlypher: Unified Vector Glyph Generation with Language Models, OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens, and Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure):
  • Emit geometry directly (autoregressive over raw path strings): VecGlypher: Unified Vector Glyph Generation with Language Models for glyphs.
  • Emit parameterized commands (custom discrete vocabulary, detokenize to JSON): OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens for Lottie animation.
  • Orchestrate existing geometry (preserve input SVG, emit only the animation layer): Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure for SVG→CSS animation.
• Specialized optimization-based vector-animation baselines (AniClipart, Livesketch) are 10–35× slower than autoregressive emission while reaching lower success rates and lower motion-alignment scores (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens Table 1).
• Data-mix matters: OmniLottie’s 0.7 native Lottie + 0.3 SVG-derived motion-augmented Lottie beats both pure-Lottie (undertrained geometry) and pure-SVG (oversimplified motion) (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens Table 2).

Recent contributions

• [2026-06-04] The Next Frontier of Visual AI Is Code: a16z position essay that names OmniLottie as the worked example for “make the source format model-native”; generalizes the vector-graphics-as-code-output thesis across SVG, HTML/CSS, Lottie, and 3D, and frames the whole cluster as instances of a Code → Render → Inspect → Revise test-time loop
• [2026-05-23] OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens: Lottie tokenizer + Qwen2.5-VL backbone + MMLottie-2M dataset; achieves 88–93% generation success on Text/TI/V-to-Lottie versus <13% for GPT-5 and 0% for Gemini3.1-Pro on Video-to-Lottie. Adds the “parameterized-command vocabulary” axis to the design space.
• [2026-05-23] Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure: Vector Prism, training-free orchestration pipeline (SVG semantic parsing → LLM/VLM planning → CSS/HTML codegen); preserves input geometry and emits only the animation layer. CVPR 2026 Highlight, fully open-source.
• [2026-05-23] VecGlypher: Unified Vector Glyph Generation with Language Models: VecGlypher, single autoregressive multimodal LLM (Gemma3 27B / Llama3.3 70B) that emits SVG <path> strings for glyphs from text or image references; two-stage SFT (39K Envato + 2.5K Google Fonts) crushes flagship LLMs by ≈2× on R-ACC and ≈83% on FID. CVPR 2026, Meta internship work.

Open questions

• Does the OmniLottie vs. VecGlypher disagreement on custom-tokenization-helpfulness generalize? A controlled experiment swapping representations across both domains would resolve whether the load-bearing variable is “amount of boilerplate in the source format” or something more domain-specific.
• At what model scale does emit-raw-geometry start to work reliably? VecGlypher shows 4B → 27B is a phase transition for SVG path quality; OmniLottie operates on a 3B Qwen2.5-VL but isn’t ablated across scales. The “below ~30B current evidence is brittle” claim from VecGlypher needs replication in other vector domains.
• Hybrid orchestrate-then-emit: can a Vector Prism-style decomposition feed a VecGlypher- or OmniLottie-style emitter for parts the orchestrator can’t reuse? No paper on file currently does this.
• Evaluation: each paper introduces its own benchmark (MMLottie-Bench, the VecGlypher glyph eval, Vector Prism’s qualitative-only assessment). A cross-paper benchmark would let the field measure progress on shared axes.
• Constrained decoding / RL with renderability rewards as a reliability fix — flagged as future work by OmniLottie (OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens §6) but not yet implemented in any filed paper.

Papers

2026-03

• OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens — OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens (published 2026-03-02)

2026-02

• VecGlypher: Unified Vector Glyph Generation with Language Models — VecGlypher: Unified Vector Glyph Generation with Language Models (published 2026-02-25)

2025-12

• Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure — Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure (published 2025-12-16)