BrowserVLM: a 0.54B GUI-grounding model for browser agents

BrowserVLM is a 0.54B-parameter vision-language model that decides where an agent should click. It scores 79.10 on ScreenSpot-V2 and 36.37 on ScreenSpot-Pro — a mean of 57.73 — which puts it ahead of UI-TARS-7B on the hard, high-resolution split at roughly a thirteenth of the size. Small enough to run beside your agent, not instead of it.

Browser and computer-use agents are usually bottlenecked by a single action: deciding where to click. Given a screenshot and an instruction like open the export menu, the agent has to resolve that intent down to a single pixel. State-of-the-art grounding vision-language models (VLMs) do this well — but the models that do it well are large, and far too slow and expensive to call hundreds of times per trajectory.

Frontier models often burn 5–12k tokens per screenshot to produce one click, and the click still may not be pixel-perfect, because those models are not fully grounded. Weaker open-source models struggle more, and get worse as their context fills up.

Molmo2 [1] and its specialised successor MolmoPoint [2] ground at ~8B parameters and are very accurate. They still carry noticeable latency and compute cost, which is a blocker for an interactive browser agent making dozens of clicks a minute.

The broader computer-use landscape leans on larger models still: closed computer-use APIs, or heavy open-source models like Kimi-VL (16B), InternVL2 (76B) and Qwen2.5-VL (72B). A 72B model needs ~144GB of VRAM at half precision — two 80GB A100s just to run inference. That rules out running the BrowserOS agent locally on anything short of a very strong GPU.

A small grounded VLM helps twice over: it keeps the agent's context from filling up with screenshot tokens, and it is markedly more accurate on small interactive elements at high resolution. So the question is obvious:

Can a <1B VLM ground clicks well enough to be of practical use to a browser agent?

We trained BrowserVLM, a 0.54B^* GUI-grounded VLM built from a Qwen-style decoder and a SigLIP2 vision tower. We pre-train it on a screen-structure-heavy mixture and post-train it on GUI pointing, which teaches it to bisect down to a specific point in an image rather than write out a coordinate — clicking at a precision of a few pixels.

On standard benchmarks BrowserVLM is competitive with 7B GUI specialists while using roughly 12× fewer parameters. We also study a recurrent-depth retrofit based on Relaxed Recursive Transformers [3], which shrinks the stored decoder by about 9% at the same effective depth.

^* All parameter counts refer to the decoder. The total model size also includes the vision tower, which adds ≈400M parameters to produce the image tokens.

Architecture

BrowserVLM borrows the standard three-stage Molmo2 stack — vision backbone → connector → decoder — fused into a single autoregressive model.

Vision. A SigLIP2 SO400M/14 encoder at 384² resolution, with SigLIP-style resizing and normalisation (pixels scaled to [−1, 1]) [4].

Tiling. Screens are high-resolution and detail-dense, so a single 384² crop discards critical information. The image is therefore tiled — a crop grid chosen to minimise upscaling, pooled into a token grid, and projected into the token stream. The crop budget is modest in pre-training (max_crops=8) and rises to 48 crops per screen in GUI post-training.

Full Qwen3 [5] and SigLIP2 configuration details are in the paper.

Pointing: patch → sub-patch → location

The pointing path is the part worth dwelling on, because the model never regresses coordinates as text. Following MolmoPoint's grounding-token scheme [2], a point is emitted as three special tokens generated coarse-to-fine — <PATCH>, <SUBPATCH>, <LOCATION> — and resolved by grounding heads that score the model's own visual tokens by cross-attention, not by the LM head. Those heads add ≈5.13M parameters.

Patch (coarse). The <PATCH> hidden state projects to a query, scored by dot product against keys from the LLM's visual-token hidden states. A softmax — plus a stop class — forms a distribution over patches, trained by cross-entropy against the target patch. The point is selected, not regressed.

Sub-patch (refine). <SUBPATCH> queries the finer-grained ViT features inside the winning patch and scores them the same way, recovering resolution lost to patch-pooling.

Location (fine). <LOCATION> places the point within the sub-patch. The chosen features feed back as the next token's input embedding, so each level is conditioned on the one before it.

Because selections score against a fixed ViT grid (14×14, roughly 4–5 px), spatial precision is resolution-independent — unlike text coordinates, whose effective resolution decays as the image gets bigger. This is what keeps a 0.54B backbone competitive on high-resolution grounding. RoPE is applied to image tokens relative to the last selected patch, which gives stable sorted point emission for counting and multi-point queries.

Depth-wise recurrent transformers

We take the core idea from Relaxed Recursive Transformers (RRT) [3], which converts depth into recurrence in two moves.

Tie. Collapse the N pretrained layers into N/B unique blocks and execute each block B times. The shared block is initialised to the average of the source layers it replaces.

Relax. Restore per-iteration freedom with a low-rank adapter, initialised from the residual between each source layer and the shared block. At step 0 the recursive model reconstructs the original full-depth model, up to rank-r truncation.

For a tied projection with shared weight W_shared and source layer ℓ, take the truncated SVD of the residual and set the additive delta:

R_ℓ  =  W_ℓ − W_shared  ≈  U_r Σ_r V_rᵀ

ΔW  =  B A,    A = Σ_r^(1/2) V_rᵀ ∈ ℝ^(r × d_in),    B = U_r Σ_r^(1/2) ∈ ℝ^(d_out × r)

so that W_shared + BA ≈ W_ℓ at initialisation. We apply this to the projectors in the block — fused QKV and gate/up/down — following the standard LoRA formulation.

On top of that base recipe we apply five refinements. Each is justified quantitatively through ablations in the paper.

1 — Prelude / coda. The most consequential choice is what gets looped. The first and last few layers of a pre-trained transformer are the most specialised — they translate between the embedding space and the residual stream — so tying them away does the most damage to the initialisation reconstruction and causes distribution drift.

We therefore keep the first P = 4 and last C = 4 layers exact and unshared, run once, and recurse only the middle N − P − C = 20 layers. With N = 28 and B = 2, those 20 layers collapse into n_core = 10 unique blocks, each executed twice. Effective depth is P + n_core·B + C = 4 + 20 + 4 = 28, realised by 18 unique blocks. This was our highest-leverage change on top of the RRT formulation. The construction mirrors the prelude / coda framing of Geiping et al. [6].

2 — Deep-residual input injection (ProjConcat). Naive recurrence forces every iteration to carry forward, inside the residual stream, any input information a later iteration might need. That costs bandwidth and parameters, because the model has to maintain circuits whose only job is moving data along.

We fix this by adopting ASURA's ProjConcat [7], a learned generalisation of the recall connection in Deep Thinking networks [8]. We re-inject the iteration's input at each loop boundary through a learned projection of the concatenation [x₀ ; x]:

x  ←  x + s_i · Proj_i([x₀ ; x]),      Proj_i : ℝ^2d → ℝ^d

The projection is zero-initialised, so nothing is injected at step 0.

3 — Attention-output adapter. RRT's relaxation, as published, adapts QKV, gate/up, and down. We add a fourth SVD-initialised additive adapter on the attention output projection, threaded as out = attn_out(a) + adapter(a) and initialised from each source layer's attn_out residual.

4 — Unshared per-iteration norms. A weight-tied block reuses the same normalisation on every pass, yet different iterations operate on differently-conditioned activations. Following ASURA, each loop iteration of the core gets its own attn_norm / ff_norm, RRT-initialised to the true source-layer norm rather than the averaged one.

5 — Layer and loop index embedding. A learned d_model vector per (loop, iteration index) pair is added to the residual at core-block entry, zero-initialised so it is a no-op at step 0. At ≈20k parameters this is the strongest single add-on: it lets otherwise identical blocks specialise by recurrence depth and position.

Parameters. Tying 28 layers into 18 unique blocks saves 157M parameters, but rank-256 adapters add 99.6M back, and 156M tied embeddings cannot be looped away. The final looped decoder is 540.6M against 596M unlooped — about 9.3% smaller.

It is worth being precise about what that buys, because it is easy to oversell. The looped decoder still performs 28 block applications per forward pass, exactly like the unlooped 28-layer model. Recurrence here does not add compute and does not change latency; it reduces stored parameters at the same effective depth, for a small accuracy cost. RRT, in this setting, is a parameter-compression retrofit — a way to ship a smaller checkpoint, not a way to make the model think longer.

It is also worth stressing that the per-iteration LoRA deltas cannot simply be folded into one shared recurrent weight. Each iteration learns a genuinely different low-rank correction to the same tied block, which is exactly why the adapters have to stay resident — and why the compression ratio lands at 9.3% rather than somewhere near the naive tying figure.

One could push the compression further by lowering the adapter rank or using more expressive, more parameter-efficient adapters. We chose not to run those experiments, for compute reasons.

Methodology and data

Our aim is a foundational screen-understanding model small enough for a browser agent, with precise clicking layered on top. That dictates a two-stage recipe: a broad multimodal pre-train, then a narrow GUI post-train with SFT to embed the <PATCH> / <SUBPATCH> pointing capability.

An effective computer-use VLM perceives the screen as structured state: what elements exist, where they sit, and what they say. ScreenVLM [9] showed that training a compact VLM to parse a full screen into a structured tag sequence — the ScreenTag representation — yields transferable structural priors that measurably improve downstream grounding.

We take that result as the basis of the design and build the mixture around screen structure and localisation. ScreenParse (25%) and pointing/counting (30%) together account for 55% of pre-training, so more than half of what the model sees is screen-structure or localisation signal. Captioning (35%) carries general visual grounding, and Tülu 4 (10%) keeps the language ability from regressing.

ScreenParse turns UI elements — bounding box, label, text — into normalised 0–500 coordinates inside an HTML-like <screentag> representation, capped at 128 elements per screen, under the prompt "Generate the screen representation for this UI." This teaches the model the layout grammar of interfaces before we ever ask it to click, and supplies useful localisation signal along the way.

Tülu 4 is there purely to stop language regression, and is aggressively filtered: empty rows, hardcoded answers, code, non-English text, and certain puzzle and verifiable-reasoning sources are dropped, as is anything whose first message exceeds ~2k tokens.

In total we pre-train on ≈1.2M samples, mixed 35% captioning / 30% pointing and counting / 25% ScreenParse / 10% Tülu 4 text, at a maximum sequence length of 4,096.

SFT post-training

A model that understands layout must still localise precisely. Rather than fold clicking into pre-training, or regress text coordinates, we add it as a focused post-training (SFT) stage on click data. During SFT we train the MolmoPoint-style grounding heads to emit the <PATCH> / <SUBPATCH> hierarchy.

This is what gives the model its precision: instead of naming a coordinate in one shot, it narrows to the slice of the image it wants in three coarse-to-fine selections, each conditioned on the last, each scored against a fixed ViT grid rather than a text vocabulary.

The click mixture is a single source. We do not train on benchmarks.

GUI post-training runs at a much larger budget than pre-training: 48 crops per screen instead of 8, and a maximum sequence length of 12,288 instead of 4,096. Dense professional UIs are the reason — small icons in high-resolution screens do not survive a coarse tiling.

Training infrastructure

The entire pipeline runs on Modal with activation checkpointing and mixed precision: both stages train under bf16 autocast, with FSDP parameters and reductions configured as float32. Learning rates follow standard VLM practice — ViT slowest, connector fastest, LLM in between — under a cosine schedule with warmup. Both stages use AdamW.

End to end, pre-training the ≈1.2M-sample mixture takes 19.4 hours in total, roughly 9.7 hours per trainer epoch, on 4× H200. GUI post-training takes ≈16 hours on 8× H200. Full launch settings and hyperparameters are in the paper.

Evaluation

We report GUI click accuracy on ScreenSpot-V2 [12] — cropped, mostly text, near-saturated at ~85–95% — and ScreenSpot-Pro [13] — full-screen, high-resolution and icon-heavy, where 7B generalists score under 2%. Each model is summarised by the mean of the two, with SS-Pro as the harder, higher-resolution test. We also run our own internal click-eval, a small handpicked benchmark built to push pointing models.

Neither ScreenSpot benchmark appears anywhere in our training data. They are held out end to end — the only click data BrowserVLM ever sees is Molmo2SyntheticPoint — so what follows is a clean evaluation, not in-distribution adaptation.

At 0.54B, BrowserVLM lands inside the 7B-specialist band. On ScreenSpot-Pro it scores 36.4, ahead of UI-TARS-7B (35.7) at roughly a thirteenth of the parameters, and within 1.7 points of UI-TARS-72B (38.1). On the mean it clears UI-TARS-2B (56.2), UGround-v1-7B (52.0) and OS-Atlas-7B (51.5).

It does not clear the dedicated 8B grounding specialists — MolmoPoint-GUI at 77.3 mean and Molmo2-GUI at 70.6 are a long way ahead — and it does not clear UI-TARS at 7B or 72B on the mean either. The gap is concentrated in ScreenSpot-V2, the saturated half of the pair, which is where a sub-1B backbone gives ground first. On the harder half it stays in the fight.

BrowserOS click-eval

This is the internal eval we use at BrowserOS to compare click models. It is a small set of handpicked tasks designed to be hard: they need precise pointing plus a bit of reasoning, not just element detection. We report the median L2 distance from the ground-truth point, in pixels. Median rather than mean, because a handful of complete misses dominates an average and tells you nothing about the typical click. The ranking below is truncated for readability.

Prompt 1 / 3“Click the bus closest to cork city centre”

Median L2 (px) — lower is better

050100300 px

Median L2 click error in pixels on our internal click-eval benchmark, lower is better. BrowserVLM ranks 10th of 19 models.

#	Model	Median L2 (px)
1	points-gui-g	2.58
2	opencua-7b	3.11
3	gui-owl-1.5-8b	4.12
4	molmopoint-gui-8b	5.52
5	mai-ui-8b	6.06
6	ui-venus-1.5-2b	6.60
7	holo2-8b	7.56
8	qwen3-vl-2b-instruct	7.69
9	groundnext-7b	8.49
10	Ours — BrowserVLM	8.89
11	qwen3-vl-4b-thinking	10.29
12	qwen3-vl-2b-thinking	11.01

18 of 19 models land under 50 px; the tail runs out to 142.70 px. Bars are log-scaled.

Show all 19 rows (+7)

For its size, BrowserVLM punches well above its weight. At 8.89 px median it lands ahead of Qwen3-VL-4B-thinking (10.29), InfiGUI-G1-7B (11.15), UI-TARS-1.5-7B (12.81), Microsoft's Fara-7B (36.58), UGround-v1-7B (43.03) and Moondream (142.70) — and it stays in single-digit pixels, the same band as the 8B grounding specialists.

Deployment

We are rolling BrowserVLM out to all BrowserOS users, wiring the click model directly into the BrowserOS harness. Any agent driving the browser gets precise interaction out of it — which matters most for frontier and local models whose own vision grounding is weak.

We are also open-sourcing the code, the checkpoints and GGUF builds, alongside a fork of llama.cpp, so the pointing model runs well on CPU and Apple Silicon (MPS) backends and not only on CUDA.

Takeaways

The result is a path to a small, deployable GUI-grounding VLM: a screen-structure-heavy pre-training mixture, a focused GUI-click post-training stage, grounding heads that select rather than regress, and a depth-wise recurrent retrofit that shrinks the stored decoder at constant effective depth — all packaged in a reproducible Modal pipeline.

The bet is that for browser automation the right point on the accuracy-versus-size curve is not the top. It is the point fast and compact enough to sit alongside the larger agents and amplify them. BrowserVLM is our first step towards a general agentic VLM that works in symbiosis with bigger reasoning models to make them more reliable.

References

1. [1] C. Clark et al. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding. Allen AI, 2026.
2. [2] C. Clark et al. MolmoPoint: Better Pointing for VLMs with Grounding Tokens. 2026.
3. [3] S. Bae et al. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA. 2024.
4. [4] M. Tschannen et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. 2025.
5. [5] Qwen Team. Qwen3 Technical Report. 2025.
6. [6] J. Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025.
7. [7] N. Gupta. ASURA: Asymptotically Universal Recursive Architecture. 2025.
8. [8] A. Bansal, A. Schwarzschild et al. Deep Thinking / DTNet. 2022.
9. [9] A. S. Gurbuz et al. Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision (ScreenVLM / ScreenParse). 2026.
10. [10] N. Lambert et al. Tülu 4. Allen AI, 2026.
11. [11] Lu et al. OmniParser for Pure Vision Base GUI Agent. 2024.
12. [12] Z. Wu et al. OS-Atlas: A Foundation Action Model for Generalist GUI Agents (introduces ScreenSpot-V2). 2024.
13. [13] K. Li et al. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. 2025.
14. [14] Y. Qin et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. 2025.
15. [15] B. Gou et al. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (introduces UGround). 2024.

Upstream Molmo2, MolmoPoint and PixMo artifacts are © Allen AI and cited here as prior work.