ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation

TL;DR

ABACUS adapts a frozen 3B-parameter unified vision-language foundation model into a single system that handles object counting, crowd counting, referring-expression counting, and count-faithful image generation, claiming SOTA across seven benchmarks without benchmark-specific training. The recipe is three targeted adaptations on top of the base model: density-aware adaptive zooming with objectness maps for spatial grounding, a boundary-aware count reward via GRPO to fix crop-edge double-counting, and a cycle-consistent GRPO where the understanding branch self-critiques generations to close the understanding-generation gap with no external annotations. The paper is interesting both as a counting-specific perception fix and as a case study of using a UMM’s own understanding branch as the reward model for its generation branch.

Key claims

• ABACUS is a unified VLM that simultaneously handles object counting, crowd counting, referring-expression counting, and count-faithful image generation, built on a 3B-parameter unified foundation model and adapted without benchmark-specific training [Abstract].
• Density-aware adaptive zooming with objectness maps is used as the spatial-grounding primitive for localization [Abstract].
• A boundary-aware count policy is trained via GRPO with the explicit goal of eliminating crop-boundary errors in count predictions [Abstract].
• Cycle-consistent GRPO: the understanding branch self-critiques the generation branch’s outputs, closing the understanding-generation gap without external annotations [Abstract].
• Reports state-of-the-art across seven benchmarks, outperforming both task-specific counting specialists and larger generalist VLMs [Abstract].

Method

ABACUS starts from an existing 3B-parameter unified foundation model (a model with coupled understanding and generation branches) and applies three adaptations. (1) For spatial grounding under high object density, an objectness map drives an adaptive zoom that crops attention to dense regions before the counting readout. (2) For boundary errors — the standard failure mode where the same instance is counted twice across crop seams or missed entirely when bisected — a GRPO policy is trained with a boundary-aware reward shaped to penalize cross-crop inconsistency. (3) For the understanding-generation gap, the model is trained with a cycle-consistent GRPO loop in which the generation branch produces an image conditioned on a target count, the understanding branch re-counts it, and the discrepancy is fed back as the reward signal. The understanding branch thereby acts as the policy’s reward model, sidestepping the need for an external count-faithfulness annotator [Abstract; project page].

Results

The paper claims state-of-the-art on seven benchmarks spanning object counting, crowd counting, referring-expression counting, and count-faithful image generation, outperforming task-specific specialists and larger generalist models. No per-benchmark numbers are exposed in the arxiv abstract or the public project page snippet retrieved at filing; code is announced but not live at the linked GitHub repo as of the Slack post [Abstract; project page].

Why it’s interesting

Counting is one of the cleanest documented VLM perception failures — Vision Language Models are Biased reported 17% average counting accuracy for frontier VLMs on counterfactual images and Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs showed GPT-4o 2D counting going from 12.0% → 38.83% with a training-free horizontal-scan-line scaffold. ABACUS attacks the same failure mode from the training-side, RL post-training angle that Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs explicitly avoided, and uses the boundary-aware reward shape to fix a sub-failure (crop-edge double-counting) that input-side scaffolds cannot address. The cycle-consistent GRPO design also extends the Unified Reward Model for Multimodal Understanding and Generation / HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing thread — using a UMM’s own understanding branch as the reward model — into the count-faithful generation setting, which is a narrower and more verifiable target than open-ended aesthetic preference. Worth tracking once code drops to see whether the boundary-aware reward and cycle-consistent GRPO recipes transfer to other counting-adjacent perception failures (instance segmentation precision, pose-keypoint completeness).

See also

• Vision Language Models are Biased — direct behavioral diagnosis of the counting failure ABACUS targets; counterfactual-image benchmark where frontier VLMs collectively score ~17% counting accuracy.
• Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs — training-free input-side fix for VLM counting via horizontal-scan-line overlays; the complementary “no training” branch of the same failure mode.
• Hidden in plain sight: VLMs overlook their visual representations — mechanistic localization of vision-centric failures to the LLM readout layer; supplies the theoretical case for why LLM-side RL post-training (ABACUS’s path) should work.
• BOP-Ask: Object-Interaction Reasoning for Vision-Language Models — the geometric-output-format analogue: 2B NVILA fine-tuned on pose-grounded synthetic data closes the frontier-vs-trained gap; same fine-tune-the-readout pattern.
• Unified Reward Model for Multimodal Understanding and Generation — earliest filed instance of “one VLM as reward model for both understanding and generation”; ABACUS narrows that to a single self-rewarding loop inside the same model.
• Unified Multimodal Models — the model class ABACUS adapts.
• Reasoning RL — GRPO post-training is the lever; cycle-consistent GRPO is a notable instance for the concept page.
• VLM Perception Failures — the failure cluster ABACUS targets.