GUI-Lens — Precise GUI Grounding via Coarse-to-Fine Cropping
GUI-Lens improves click localization on dense, high-resolution interfaces without additional model training. Coordinate Priming converts OCR text and detected UI components into position references for an off-the-shelf vision-language model.
The model selects progressively finer crops or proposes a click based on the instruction and current view. Crops and clicks are checked against the instruction, and accepted local click coordinates are mapped back to the original screen.
GUI-Lens provides visual grounding for desktop agent workflows in OpenProgram. The repository provides the implementation and ScreenSpot-Pro evaluation instructions.