GUI agents
PulseAugur coverage of GUI agents — every cluster mentioning GUI agents across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
New framework boosts GUI agent training efficiency by 60%
Researchers have developed a novel two-stage adaptive training framework designed to enhance the efficiency of training GUI agents. This method addresses the inefficiency of repeatedly rolling out complete trajectories …
-
FocusGUI adapter boosts GUI agent accuracy by prioritizing multimodal evidence
Researchers have introduced FocusGUI, a novel adapter designed to enhance the performance of graphical user interface (GUI) agents. This system dynamically selects relevant multimodal evidence, such as screenshots and a…
-
New frameworks aim to improve GUI agent reliability and stability
Researchers have introduced two new frameworks aimed at improving the reliability and stability of GUI agents. TaskVM, through its task-state governance paradigm, allows agents to execute workflows with verifiable and r…
-
New GUI-Hopper method enhances mobile agents with deeplinks
Researchers have developed a new method called GUI-Hopper to improve the efficiency of mobile GUI agents. This approach combines app-native deeplinks with traditional GUI actions like taps and swipes. By using deeplinks…
-
New TRACE framework improves GUI agent efficiency with visual token pruning
Researchers have developed TRACE, a novel framework designed to enhance the efficiency of GUI agents. TRACE addresses the challenges of increasing latency and memory usage caused by high-resolution screenshots in agent …
-
TRACE framework enhances GUI agent efficiency with novel evidence ordering
A new framework called TRACE has been developed to improve the efficiency of GUI agents by reducing latency and memory usage. TRACE achieves this by ranking visual evidence based on its future utility and diversity, res…
-
New system engineers SOPs for professional GUI agent use
Researchers have developed OmegaUse-SOP, a novel system designed to engineer Standard Operating Procedures (SOPs) for professional computer use. This human-in-the-loop system transforms expert demonstrations into reusab…
-
New Chameleon framework exploits GUI agent vulnerabilities in dynamic environments
Researchers have developed a new attack framework called Chameleon designed to exploit vulnerabilities in GUI agents operating in dynamic online environments. Existing methods for Environmental Injection Attacks (EIAs) …
-
New benchmark AnTrap reveals universal vulnerability in Android GUI agents
Researchers have developed AnTrap, a new benchmark designed to evaluate the robustness of Android GUI agents against runtime anomalies. The benchmark injects dynamic perturbations into agent execution trajectories, cate…
-
New KV cache compression techniques aim to boost LLM long-context performance
Researchers are developing new methods to compress the key-value (KV) cache in large language models, a major bottleneck for long-context inference. Minima-KV uses a mixed-format approach, storing recent pages in FP8 an…
-
New Gated Hindsight Distillation enhances GUI agent training
Researchers have developed a new training technique called Gated Hindsight Distillation (GHD) to improve the performance of GUI agents. GHD utilizes future screenshots as privileged information during training, allowing…
-
New Android GUI Agent Vulnerability Exploits Multimodal Model Weaknesses
Researchers have identified a novel security vulnerability in Android GUI agents powered by large multimodal models. These agents, designed to perceive screen content and inject inputs, are susceptible to "Action Rebind…
-
New red-teaming method exploits GUI agent vulnerabilities
Researchers have developed a new black-box red-teaming method called Semantic-level UI Element Injection to test the robustness of GUI agents. This technique overlays harmless UI elements onto screenshots to misdirect a…
-
New metric reveals GUI agents prioritize structure over pixels
Researchers have developed a new metric called the Perception-Fusion Gap to diagnose how multimodal GUI agents form beliefs about their interface state. This metric measures the extent to which an agent relies on visual…
-
New GUIDE framework reduces domain bias in GUI agents using video retrieval
Researchers have developed GUIDE, a novel framework designed to mitigate domain bias in GUI agents. This plug-and-play system leverages real-time web video retrieval and an automated annotation pipeline to equip agents …
-
New GAIA system trains critic models to improve GUI agent performance
Researchers have developed GAIA, a data flywheel system designed to improve the performance of GUI agents by training an Intuitive Critic Model (ICM). This ICM evaluates the correctness of an agent's actions, selecting …
-
VisCritic framework enhances GUI agents with visual state comparison
Researchers have introduced VisCritic, a novel visual process reward framework designed to enhance the performance of GUI agents. Unlike previous methods that rely solely on textual reasoning, VisCritic directly compare…
-
New EVA framework evolves semantic attacks on GUI agents
Researchers have developed EVA, an evolutionary framework designed to identify semantic vulnerabilities in GUI agents powered by multimodal large language models (MLLMs). This method focuses on manipulating the semantic…
-
StainFlow improves GUI agent training with novel reward model
Researchers have introduced StainFlow, a novel process reward model designed to enhance the training of GUI agents. This method addresses the sparsity of feedback in reinforcement learning by providing finer-grained tra…
-
New DragOn dataset boosts GUI agent drag-and-drop capabilities
Researchers have introduced DragOn, a new benchmark and dataset designed to improve the performance of GUI agents in handling drag-based interactions. The dataset includes 286,000 training screenshots and 3.5 million tr…