GUI agents are rapidly evolving and are beginning to appear in computer and mobile applications for automated use, enabling end-to-end automation through direct perception of and interaction with on-screen interfaces. However, these agents may frequently access user interfaces containing highly sensitive personal information, and in practice screenshots are often transmitted to remote models for inference, creating substantial privacy risks during daily use. These risks are particularly severe and distinctive in GUI agent workflows: compared with other visual carriers, GUIs expose richer and more directly accessible private information, and privacy risks are highly dependent on interaction trajectories and contextual continuity, requiring assessment across sequential scenes. Based on these observations, we propose \textbf{GUIGuard}, a general three-stage framework for privacy-preserving GUI agents consisting of (1) privacy recognition, (2) privacy protection, and (3) task execution under protection. To effectively analyze privacy risks in GUI agent workflows and to rigorously evaluate GUIGuard and future protection methods, we further construct \textbf{GUIGuard-Bench}, a cross-platform benchmark containing 630 GUI agent trajectories with 13{,}830 screenshots, annotated with region-level privacy grounding as well as fine-grained labels of risk level, privacy category, and task necessity. Evaluations on GUIGuard-Bench reveal that existing agents exhibit limited privacy recognition, with the sate-of-the-art model achieving end-to-end accuracies of only 13.3\% on Android and 1.4\% on PC; meanwhile, under privacy protection, task-planning semantics can still be maintained to an acceptable level, where closed-source models show substantially stronger semantic consistency than open-source ones. Finally, the case study on the MobileWorld online dataset shows that carefully designed privacy protection strategies can strike a better balance, achieving higher task accuracy while still preserving adequate privacy. Our results highlight privacy recognition as a critical bottleneck for practical GUI agents and point to concrete directions for building safer, privacy-aware GUI agent systems.
Privacy Detection Module. This stage answers “what should be protected.” It produces a structured privacy representation that guides downstream protection, including (i) spatial localization of sensitive regions (e.g., regions or bounding boxes), (ii) semantic categories and risk levels, and (iii) whether each item is necessary for the current task. Detection combines textual signals from the interaction context to maintain step consistency. At the system level, this module prioritizes high recall and auditability, preferring conservative triggers over the irreversible risk of missing critical sensitive content.
Privacy Protection Module. This stage answers “how to protect.” It maps detected regions to a configurable set of sanitization operators and policy compositions, making the trade-off between privacy and interface fidelity explicit. The system may use masking, text replacement, and similar techniques based on risk and task requirements. Auditable policies should sanitize sensitive information sufficiently to prevent reconstruction, while avoiding over-sanitization that leads to unrecognizable elements, grounding failures, or plan drift.
Task Execution Module. This stage answers “whether tasks remain solvable after privacy protection.” Under a privacy-protected interface and task context, the system generates executable action plans and interacts with a real GUI. Unlike conventional end-to-end GUI agents, the execution process is explicitly constrained by partial observability, where sensitive visual information may be masked or removed. As a result, the agent must reason over historical trajectories and residual visual cues to infer and execute subsequent actions. Effectiveness is evaluated using end-to-end task success rate, semantic consistency of planning outputs, and grounding accuracy. By jointly considering the quality of privacy detection and the effectiveness of protection strategies, these criteria establish a quantifiable and comparable closed-loop framework for evaluating privacy-preserving GUI agents.
The dataset structure is illustrated above. It consists of 240 trajectories (4,080 screenshots) collected from real-world terminal platform scenarios and 390 trajectories (8,587 screenshots) generated using image-generation models. Each screenshot includes textual logs of the agent’s interactions. Privacy-related regions are annotated with three risk levels and six semantic categories, and task-required privacy is explicitly marked to enable benchmarking across recognition, protection, and execution phases.
GUI-Trajectory-Virtual alternates between an image-generation model and the GUI agent. Starting from the initial task and a real screenshot, the generator synthesizes the next-step GUI state. The resulting screenshot and task description are fed to the agent to produce the next action plan, which—together with historical screenshots—is used to synthesize the subsequent screenshot. This loop continues until the task completes, and all generated plans plus screenshots are collected into the final dataset.
Privacy recognition results on GUIGuard-Bench for both PC (blue) and Android (red) devices: left shows binary privacy-detection accuracy (whether a screenshot contains any sensitive content), middle shows privacy recall (percentage of ground-truth private elements retrieved), and right reports the overall end-to-end accuracy that requires correct detection alongside all fine-grained labels (risk level, category, and task necessity).
Fine-grained privacy label recognition on GUIGuard-Bench. For private elements that pass text matching and are correctly localized (IoU ≥ 0.6), we report the accuracy of predicting risk level, privacy category, and task necessity across VLMs on PC (blue) and Android (red).
The planner is the module most relied upon by modern agents for remote services; thus, variations in its performance directly reflect changes in execution ability and privacy-protection fidelity. The grounding module is removed and subsequent screenshots are treated as step outcomes, enabling direct evaluation of planner outputs. Planning traces produced with privacy-masked screenshots are compared against the agent’s own unprotected baselines via a self-comparison protocol that reuses identical inputs and task contexts. An LLM-as-Judge scores semantic consistency at each step to quantify the fidelity of privacy protection.
| Category | Model | Black Mask | Mosaic Mask | Random Blocks | Text Box Replace | Android Avg. | PC Avg. | Overall Avg. |
|---|---|---|---|---|---|---|---|---|
| GUI Agent | UI-Tars-1.5-7B | 1.16 | 1.44 | 0.88 | 1.01 | 1.27 | 0.95 | 1.11 |
| Fara-7B | 2.02 | 1.94 | 2.11 | 1.95 | 2.34 | 1.69 | 2.04 | |
| GUI-Owl-7B | 2.56 | 2.50 | 2.85 | 2.32 | 2.63 | 2.47 | 2.55 | |
| Open-source VLM | Qwen3-VL-235B-A22B | 2.71 | 2.92 | 2.63 | 2.63 | 2.92 | 2.52 | 2.73 |
| Closed-source VLM | Claude-Sonnet-4.5 | 2.87 | 2.74 | 3.00 | 3.03 | 3.44 | 2.42 | 2.96 |
| GPT-5.2 | 3.33 | 3.30 | 3.44 | 3.17 | 3.33 | 3.28 | 3.31 | |
| Gemini-3-Pro | 3.32 | 3.42 | 3.40 | 3.22 | 3.45 | 3.24 | 3.35 | |
| Fidelity of Protection Methods | 2.57 | 2.61 | 2.62 | 2.48 | — | — | — | |
@article{guiguard2025,
title={GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents},
author={Author, First and Author, Second and Author, Third},
journal={arXiv preprint arXiv:2501.xxxxx},
year={2025}
}