Edit Banana: image-to-editable-DrawIO conversion via SAM3 + VLM + OCR

TL;DR

Edit Banana is an open-source pipeline that converts static diagram images (flowcharts, architectures, technical schematics, formulas) into fully editable DrawIO XML by stitching together a fine-tuned SAM 3 mask decoder, a multimodal LLM for fixed multi-round scanning, local OCR (Tesseract / PaddleOCR) for text, and Pix2Text for LaTeX formula recognition. The repository is BIT-DataLab’s reference implementation behind the hosted www.editbanana.net service, with a credit-based user system and a global-lock + LRU-cache concurrency layer for shared GPU access. It is a deployment artifact, not a research paper — no benchmarks, no quantitative claims — so its value here is as a concrete recipe for “structured-perception VLM tool” on the inverse direction of the recent academic-illustration-generation work.

Key claims

The pipeline is four stages: input image → fine-tuned SAM 3 segmentation → parallel text extraction (Tesseract bounding boxes + Pix2Text on high-res crops for LaTeX formulas) → DrawIO XML emission that merges spatial masks with OCR results [§Architecture Pipeline].
The segmentation backbone is a fine-tuned SAM 3 mask decoder rather than off-the-shelf SAM3, suggesting BIT-DataLab specialized SAM3 on diagram element masks rather than natural-image objects [§Key Features].
The text path uses a “crop-guided” strategy — text/formula regions detected by OCR are re-cropped at high resolution and re-sent to Pix2Text for LaTeX conversion, rather than running a single OCR pass over the whole image [§Key Features].
A multimodal LLM performs “fixed multi-round VLM scanning” to guide extraction; the README does not specify which VLM or how many rounds [§Key Features].
The hosted service implements per-user credit accounting, a global lock for thread-safe GPU access, and an LRU cache that persists SAM3 image embeddings across requests to amortize the dominant cost [§Key Features].
The README explicitly warns that the GitHub repo trails the hosted www.editbanana.net service in features and performance [Top of README warning callout].
Roadmap lists “Intelligent Arrow Connection” (associating arrow endpoints with target shapes) as the next in-progress item — confirming this as the current weak spot of the pipeline [§Development Roadmap].

Method

The system inverts the usual generative direction: instead of producing a diagram from a description, it takes a static raster diagram and reconstructs the underlying editable representation. SAM 3 produces per-element masks; in parallel a local OCR engine (Tesseract default; PaddleOCR 3.2.2 as alternative for mixed text) locates text bounding boxes, and Pix2Text runs on high-resolution crops of math regions to convert formulas to LaTeX. A VLM is invoked in multiple rounds to guide element extraction — the README is non-specific on the prompt structure or the VLM identity. A merge step combines the geometric layout (mask centroids, bounding boxes, arrow strokes) with the textual content into DrawIO’s XML schema, producing a file where every shape, arrow, and text label is an independently selectable, restyleable element.

The deployment piece is non-trivial: SAM 3 inference is the GPU-heavy step, so the server uses a global lock to serialize GPU access and an LRU cache to keep image embeddings hot across the OCR + VLM-scan rounds that happen for the same image. The credit system prevents free-tier abuse and is the productization layer over the open-source pipeline.

Results

No benchmarks. The README shows four qualitative input/output pairs (basic flowchart, multi-level architecture, technical schematic, scientific formula) and two interaction GIFs (manual repair, save-local). The roadmap candidly lists “Intelligent Arrow Connection” as in-development, implying that current arrow-to-shape association is heuristic and brittle.

Why it’s interesting

This is the inverse-direction sibling of PaperBanana: Automating Academic Illustration for AI Scientists — PaperBanana generates academic methodology diagrams from text; Edit Banana recovers editable diagrams from rendered images. Both projects share the “banana” naming convention (a wink at the Nano-Banana-Pro image model lineage) and both rely on a VLM-in-the-loop for structured perception. The interesting tension: PaperBanana’s reported failure mode was the Critic VLM missing connection-topology errors (PaperBanana: Automating Academic Illustration for AI Scientists §Failure Cases); Edit Banana’s roadmap explicitly flags arrow→shape association as the next problem to solve — independent confirmation that fine-grained diagram-graph perception is the open problem for VLMs even when they’re given dense pixel-level inputs. The deployment recipe (global lock + LRU embedding cache for shared SAM3 inference) is also a concrete reference for any team productizing a SAM-based service. For Luma, the most reusable piece is the SAM3-fine-tuned-on-diagrams + cropped-OCR composition: a clean template for any “perception over structured graphics” task where the OCR+segmentation+VLM seams matter more than the individual model quality.