ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMs

TL;DR

ArtifactLens is a low-supervision system for detecting visual artifacts in generated images (distorted hands, warped objects, broken anatomy, bad interactions). The headline claim is that pretrained VLMs already encode the perception needed for artifact detection — what was missing is the scaffolding. With a multi-component architecture combining in-context learning and text-instruction optimization, the system reaches SOTA on five human-artifact benchmarks using only a few hundred labeled examples per artifact category, several orders of magnitude less than the prior fine-tuning recipes. The framing matters: as image generators evolve, labeled-artifact datasets become stale, so a low-label detector is what keeps reward models and benchmarks tracking the frontier.

Key claims

• Pretrained VLMs already encode the knowledge needed to detect synthetic-image artifacts, and can be unlocked using only a few hundred labeled examples per artifact category — orders of magnitude less than prior fine-tuned detectors that use tens of thousands of labels [Abstract].
• ArtifactLens reaches state-of-the-art on five human-artifact benchmarks, the first evaluation that spans multiple artifact datasets at once rather than a single benchmark [Abstract].
• The required scaffolding is a multi-component architecture combining in-context learning and text-instruction optimization, with novel improvements to each [Abstract].
• The same scaffolding generalizes beyond human artifacts to object morphology, animal anatomy, and entity-interaction artifacts, and to the distinct task of AIGC (AI-generated content) detection [Abstract].

Method

ArtifactLens is presented as a system on top of a frozen pretrained VLM, not a new fine-tuned detector. The scaffolding has two parts the abstract calls out explicitly:

1. In-context learning — a small set (hundreds, not tens of thousands) of labeled artifact examples per category are shown to the VLM as in-context demonstrations rather than burned into weights via fine-tuning. The paper claims novel improvements to the ICL recipe but the abstract does not disclose specifics.
2. Text-instruction optimization — the textual prompt / instruction given to the VLM is itself optimized (the abstract again does not say whether this is gradient-based, search-based, or LM-driven). The combination of optimized text prompts plus a small bank of demonstrations is what unlocks the latent artifact-detection capability.

The benchmark protocol is explicitly cross-dataset: five human-artifact benchmarks evaluated jointly, with the same scaffolding stress-tested on object-morphology, animal-anatomy, entity-interaction, and AIGC-detection tasks. Quantitative numbers and the architectural diagram are in the full paper; the retrieved abstract surfaces only the headline claims.

Results

The abstract reports SOTA on five human-artifact benchmarks at orders-of-magnitude lower labeling cost than prior fine-tuned detectors, with cross-task generalization to four additional artifact families. Exact numerical deltas, the VLM backbones tested, and the per-benchmark breakdown are not in the retrieved abstract.

Why it’s interesting

The wiki has been tracking VLM-as-evaluator papers as a sequence of what does the VLM emit (scalar tier, structured CoT reward, discrete pick, per-sample QA) — see VLM-as-Evaluator. ArtifactLens is interesting because it shifts the bottleneck from what the VLM emits to how little supervision is needed to unlock it: rather than fine-tuning a Qwen3-VL or InternVL-3 on tens of thousands of labeled images (the ERNIE-Image-Aes / UnifiedReward-Flex recipe), the claim is that a pretrained VLM plus optimized prompt + a few hundred ICL exemplars is enough. If it replicates, that’s a substantial cost reduction for keeping artifact-detection benchmarks and RLHF rewards in sync with rapidly-evolving generators. It is also the natural complement to Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection (X-AIGD), which is the dataset/benchmark side of the same problem — X-AIGD provides the pixel-level categorical labels; ArtifactLens argues you only need a small subset of them to build a working detector. Open question: whether ICL-with-hundreds-of-examples is as robust to generator drift as fine-tuning-on-tens-of-thousands — drift is exactly the failure mode prior labeled-dataset detectors had.

See also

• VLM-as-Evaluator — the parent concept; ArtifactLens is a low-label recipe for the same VLM-as-judge primitive used by ERNIE-Image-Aes and UnifiedReward-Flex.
• Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection — X-AIGD; the dataset/benchmark side of the same problem (pixel-level categorical artifact annotations) — ArtifactLens argues that only hundreds of those labels are needed to train a working detector.
• Unified Personalized Reward Model for Vision Generation — UnifiedReward-Flex; per-prompt context-adaptive VLM reward trained SFT-then-DPO on 90K traces. Sits on the opposite end of the supervision axis from ArtifactLens (90K vs hundreds).
• ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization — ERNIE-Image-Aes; static-rubric scalar scorer fine-tuned on Swiss-tournament labels. Cited here for the same “fine-tune a VLM judge on a curated label set” recipe ArtifactLens is arguing against.