Manan Arora
← All projectsResearch · 3D visual grounding

LilSeeGround

Can a vision-language model small enough to ride on a robot find “the chair left of the TV” in a 3D scan? I swapped the 72B model inside a state-of-the-art pipeline for 2B and 4B ones. The 4B model did better.

Role
Led the project · team of 3
With
Vu Nguyen, Sarang Rajeev
When
Winter 2026 · University of Michigan
Stack
Qwen3-VL · vLLM · PyTorch3D
Rendered point cloud of scene0015_00 with numbered objects
Query

there is a beige wooden desk. placed on the side of the wall.

4BQwen3-VL-4B picked object #28IoU 1.00 with the true box✓ right
2BQwen3-VL-2B picked object #34IoU 0.00 with the true box✗ wrong
1 / 8
Real outputReal answers from both models on ScanRefer validation scenes. Every object in the render carries a number; the rings show which number each model chose. Green ring = right (IoU ≥ 0.25 with the true box). Use ← → to step through.
01The question

Is the accuracy cliff about size, or about age?

3D visual grounding means pointing at the right object in a 3D scene from a sentence: “the brown ottoman in front of the sofa.” A home robot or an AR headset needs it before it can act on an instruction.

SeeGround (CVPR 2025) does this without training any 3D model. It renders the scene into an image, numbers every object, and asks a vision-language model which number matches. It’s clever, and it runs on Qwen2-VL-72B: about 144 GB of GPU memory, four A100s, a kilowatt or two. No robot carries that.

SeeGround’s own ablation shows that shrinking the model hurts: a 7B model from the same family loses about 11 points. But that compares models of the same generation. I wanted to know whether a newer, much smaller model could close the gap.

144 GBGPU memory for the 72B backbonebfloat16, about 4× A100 80 GB
64 GBmemory on the biggest Jetson AGX Orinthe edge budget
−10.8points of Acc@25 going 72B → 7BSeeGround’s own ablation, same model family
02Method

Swap one part, hold everything else fixed

  1. 01Parse the queryAn LLM pulls out the target object and the anchor it’s described relative to.
  2. 02Pick a viewpointA virtual camera is placed to look at the anchor, the way a person would.
  3. 03Render with IDsThe 3D point cloud becomes a 2D image, with every object stamped with a number.
  4. 04VLM picks a numberImage + a table of object positions go to the VLM, which answers with an ID.72B → 4B / 2B
SeeGround’s four stages, redrawn. Only stage 4 changes.

I replaced Qwen2-VL-72B with Qwen3-VL at 2B and 4B, served with vLLM. Prompts, viewpoint selection, rendering and the object table (built from Mask3D instance predictions) stayed identical, so any change in accuracy comes from the backbone.

Why Qwen3-VL? It’s a drop-in successor with the same input format, and it changes how the model encodes position: Interleaved-MRoPE spreads height, width and time signals across attention heads instead of splitting them. SeeGround’s whole trick is a picture with small numbers at precise pixel locations, so finer spatial encoding should matter more here than raw size.

03Results

The 4B model beats the 72B baseline

47.2Acc@25 on Nr3D for the 4B modelvs 44.1 for the 72B baseline
+4.1points on “Multiple” queriesthe hard split: several look-alike objects in the room
18×fewer parameters4B vs 72B; the 2B model is 36× smaller
303540455025102050100parameters (billions, log)Acc@25 on Nr3D (%)Qwen2-VL-7B: 33.3Qwen2-VL-7BInternVL2-8B: 34.3InternVL2-8BInternVL2-26B: 38InternVL2-26BQwen2-VL-72B: 44.1Qwen2-VL-72BQwen3-VL-2B: 42.41Qwen3-VL-2BQwen3-VL-4B: 47.23Qwen3-VL-4B
SeeGround paperQwen3-VL (this project)
Real outputAccuracy against model size (log scale). Grey points are SeeGround’s published backbones; green are mine. Both new models sit well above the trend the older models set.
0 GB50 GB100 GB150 GBQwen2-VL-72B: 144 GBQwen2-VL-72B144 GBInternVL2-26B: 52 GBInternVL2-26B52 GBInternVL2-8B: 16 GBInternVL2-8B16 GBQwen2-VL-7B: 14 GBQwen2-VL-7B14 GBQwen3-VL-4B: 8 GBQwen3-VL-4B8 GBQwen3-VL-2B: 4 GBQwen3-VL-2B4 GBJetson AGX Orin, 64 GB
Real outputGPU memory at bfloat16, no quantization. Both new models fit inside a Jetson AGX Orin with room to spare.
Nr3D grounding accuracy (%) at IoU 0.25 and 0.50. † from the SeeGround paper. Unique: one object of that class in the room. Multiple: several.
BackboneParamsVRAMUnique@25Unique@50Multiple@25Multiple@50Acc@25Acc@50
Qwen2-VL-7B †7B14 GB————33.30—
InternVL2-8B †8B16 GB————34.30—
InternVL2-26B †26B52 GB————38.00—
Qwen2-VL-72B †72B144 GB75.7068.9034.0030.0044.1039.40
Qwen3-VL-2B (ours)2B4 GB73.0368.5432.0729.2242.4139.15
Qwen3-VL-4B (ours)4B8 GB74.1667.9838.1434.3547.2342.84
04Where it fails

Crowded rooms and look-alike objects

Both models still miss when a room is full of near-identical things and the sentence leans on fine spatial detail. And the smaller model isn’t strictly worse: switch the demo above to “2B right” to see queries the 2B model got and the 4B model didn’t.

Rendered point cloud with numbered objects

“there is an office chair. placed on the side of the wall.”

Both wrong · IoU 4B 0.00 · 2B 0.00
Rendered point cloud with numbered objects

“there are two black chairs situated between six brown chairs and a black couch. this black chair is next to the black couch. it appears to be leather. it is black. there is a snack machine on the opposite wall.”

Both wrong · IoU 4B 0.01 · 2B 0.01

Working on perception for robots or edge devices? I’d love to compare notes.

Email meLinkedIn ↗GitHub ↗Résumé ↗