GRAFT — Region-Grounded Fine-tuning of SigLIP
Taught spatial grounding into a frozen SigLIP-B/16-384 with a LoRA adapter (295K params, 0.14%) and a FILIP-style region loss on phrase→bbox pairs. The dominant gain came not from the loss or capacity but from eliminating train/eval representation mismatches (MaskCLIP bypass + EOS alignment) — raising Flickr30k Pointing Game from 14.74% to 20.76 ± 0.41% (3-seed, 6× lower variance) and 26.16% test PG at 5 epochs. Florence-2 auto-labels reach 23.35% with zero human boxes.
- +11.4pp Pointing Game
- 295K params (0.14%)
- 6× lower variance
- Florence-2 auto-labels
PyTorch · SigLIP · LoRA / PEFT · FILIP · Flickr30k · W&B