ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

TL;DR

ENEAS is a unified, text-promptable segmentation system from Speridlabs that runs two modes from the same method: precise long-horizon instance tracking, and open-concept semantic discovery. It is framed against three named failure modes of current text-promptable segmenters (including SAM 3: Segment Anything with Concepts): temporal hallucinations (missing target-absence), spatial fragmentation at extreme close-up, and semantic misclassification of doppelgangers (statues, paintings, reflections). The tracker extends the geometrically-robust SeC architecture with a text-prompting adapter and reuses its temporal memory to hold identity through disappearance; the discovery branch stacks a fast visual-embedding matcher with a conditional VLM verification layer invoked only for ambiguous candidates. Reported gains over SAM 3 on SA-Co/VEval are modest on the SA-Co-native HOTA/DetA numbers but larger on TETA (+1.21 pts on Instance Tracking). Code, weights, and demo are Apache 2.0.

Key claims

ENEAS targets three named failure modes of state-of-the-art text-promptable segmenters (SAM 3 included): temporal hallucinations (failing to report target absence when an object leaves the view), spatial fragmentation (segmenting local textures instead of the whole object at extreme close-up), and semantic misclassification (segmenting statues, paintings, or reflections as the target instance) [Abstract].
A single method runs two modes from one interface: (a) precise tracking and high-quality segmentation of a unique instance across a sequence, (b) open-concept discovery of every instance matching a text query, resolved by a downstream semantic verification layer [Abstract].
The tracker extends SeC (previously limited to point interactions) with a text-prompting adapter and leverages SeC’s temporal memory to hold the target through disappearance without drifting to distractors, keeping it whole when it fills the entire view [Abstract].
The discovery branch is a two-stage cascade: high-speed visual-embedding matching handles clear cases, and a VLM-based semantic refinement is invoked only for ambiguous candidates — filtering ontological doppelgangers (statue/painting/reflection) that visual-only similarity cannot distinguish, while keeping average-case latency low [Abstract].
On SA-Co/VEval instance tracking (SAM 3’s own official evaluator, identical settings), ENEAS scores HOTA 26.70 / DetA 7.92 / AssA 90.77 / LocA 87.86 / TETA 17.86 vs SAM 3’s 26.51 / 7.84 / 90.18 / 89.22 / 16.65 — a +1.21-point TETA lead, with SAM 3 marginally stronger on LocA [Project page Table 1].
On SA-Co/VEval semantic discovery, ENEAS scores HOTA 9.23 / DetA 6.16 / AssA 14.28 / LocA 74.25 / TETA 9.53 vs SAM 3’s 9.19 / 6.21 / 13.85 / 75.38 / 9.10 — a +0.43-point TETA lead, again with SAM 3 marginally stronger on LocA and DetA [Project page Table 2].
Designed with 3D reconstruction in mind — a single misclassified distractor corrupts the reconstructed asset, so ontological discrimination of true instance vs doppelganger is treated as a primary requirement rather than an accessory [Abstract].
Code and models released at github.com/speridlabs/eneas with a Hugging Face demo Space, both under Apache 2.0 [Slack post; Project page].

Method

ENEAS composes two frozen or fine-tuned backbones behind a unified text-promptable interface. For instance tracking, the SeC (Segmentation-then-Consistency) architecture — geometrically robust to viewpoint and scale changes and originally driven by user point interactions — is extended with a text-prompting adapter so the same tracker can be initialized from a natural-language name. SeC’s temporal memory carries the identity across occlusion and re-emergence: when the target leaves the frame and returns, the memory recovers it as the same instance rather than instantiating a new track.

For semantic discovery, ENEAS runs a two-stage cascade. A fast visual embedding matcher scores candidate regions against the concept prompt, resolving obvious matches at low latency. Only ambiguous candidates — those where visual similarity is inconclusive — trigger a conditional VLM refinement step that applies semantic reasoning to distinguish, e.g., a real dog from a marble statue of a dog, a painting of a car from a car, a mirror image of a person from the person. This gates the expensive semantic call on the cases that actually need it.

The system takes a natural-language prompt and a set of frames — either an ordered video or an unordered image collection — and returns per-instance masks with stable IDs.

Results

Both tables below use the official SAM 3 evaluator with identical settings on a broad SA-V Test selection from SA-Co/VEval, split by prompt into unique-object (instance tracking) and multiple-instance (semantic discovery) subsets.

SA-Co/VEval Instance Tracking [Project page Table 1]:

HOTA: 26.70 (ENEAS) vs 26.51 (SAM 3) → +0.19
DetA: 7.92 vs 7.84 → +0.08
AssA: 90.77 vs 90.18 → +0.59
LocA: 87.86 vs 89.22 → −1.36 (SAM 3 wins)
TETA: 17.86 vs 16.65 → +1.21

SA-Co/VEval Semantic Discovery [Project page Table 2]:

HOTA: 9.23 vs 9.19 → +0.04
DetA: 6.16 vs 6.21 → −0.05 (SAM 3 wins)
AssA: 14.28 vs 13.85 → +0.43
LocA: 74.25 vs 75.38 → −1.13 (SAM 3 wins)
TETA: 9.53 vs 9.10 → +0.43

The pattern is consistent: ENEAS wins on TETA (Track Every Thing Accuracy, which balances detection, association, and localization more evenly than HOTA) and on AssA (association across frames — the direct beneficiary of SeC’s temporal memory), while SAM 3 remains marginally ahead on LocA (per-mask localization quality on already-detected instances). No numbers are reported on the target failure modes (target-absence detection, close-up integrity, doppelganger discrimination) — those are demonstrated qualitatively in the paper and project page video.

Why it’s interesting

ENEAS is an interesting first datapoint in a pattern: for perception tasks where the required output is identity-stable object masks under adversarial framing (extreme close-up, exit-and-return, semantic doppelgangers), stacking a specialized geometric tracker with a conditional-VLM verifier appears to close the “visual-only vs ontological” gap that even 848M-parameter SAM 3 leaves open. The named failure modes echo the VLM Perception Failures cluster: statues/paintings/reflections being segmented as their referents is the segmentation counterpart to the counterfactual-prior failures (Vision Language Models are Biased) and the pixel-precise-output collapse (BOP-Ask: Object-Interaction Reasoning for Vision-Language Models) that page documents — visual-only models cannot distinguish ontological identity from visual similarity without a semantic head to arbitrate. ENEAS’s conditional-VLM cascade is a concrete instance of the VLM-as-Evaluator pattern applied to segmentation post-hoc rather than to reward-scoring or generation-time verification. The headline numeric gains over SAM 3 are small on standard SA-Co metrics; the case for ENEAS rests on whether the failure-mode-specific qualitative claims (which are not quantified against SAM 3 on a dedicated benchmark) hold in production 3D-reconstruction pipelines, where a single misclassified doppelganger does corrupt the asset.