graphical user interface
PulseAugur coverage of graphical user interface — every cluster mentioning graphical user interface across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
New PalmClaw framework enables LLM agents to run natively on mobile phones
Researchers have developed PalmClaw, an open-source framework enabling large language model (LLM) agents to run natively on mobile phones. Unlike existing systems that rely on graphical user interface interactions, Palm…
-
Instagui tool generates GUIs from CLI help text
A new tool called Instagui has been developed to bridge the gap between command-line interfaces (CLIs) and graphical user interfaces (GUIs). Instagui analyzes the --help output of CLI commands to automatically generate …
-
New benchmarks and methods improve AI agent uncertainty quantification
Researchers have developed new methods for quantifying uncertainty in AI agents that interact with graphical user interfaces (GUIs) and in vision-language-action models (VLAs) used in robotics. The first study, "Argus,"…
-
Survey maps multimodal AI for code generation from visual inputs
A new survey paper published on arXiv explores the emerging field of Multimodal Code Intelligence. This field focuses on AI models that can understand and generate code based on visual inputs like screenshots, charts, a…
-
Survey maps multimodal code intelligence systems and proposes future research directions
This survey paper categorizes and analyzes multimodal code intelligence systems, which generate and reason with code based on visual inputs. It organizes existing approaches into four domains: Graphical User Interface, …
-
New benchmark evaluates AI agents on mixed mobile device interactions
Researchers have introduced PhoneHarness, a new benchmark and execution framework designed to evaluate AI agents that interact with mobile devices. Unlike previous methods that focused solely on GUI controls, PhoneHarne…
-
New research tackles safety and efficiency in computer-use agents · 6 sources tracked
Recent research is exploring the safety and efficiency of computer-use agents (CUAs). One paper introduces MisActBench and a guardrail called DeAction to detect and correct misaligned actions, significantly reducing att…
-
Fudan, Tongyi Lab unveil ToolCUA for agents choosing between GUI and tools
Researchers from Fudan University and Tongyi Lab have developed ToolCUA, a new training paradigm for agents that can effectively utilize both graphical user interface (GUI) operations and tool calls. Experiments reveale…
-
ScreenSearch system improves AI agent exploration of desktop GUIs
Researchers have developed ScreenSearch, a novel system designed to improve the exploration of desktop graphical user interface (GUI) states for AI agents. The system addresses the challenge of partial observability, wh…
-
EVE framework launches open-source LLMs for Earth Intelligence
Researchers have developed EVE, an open-source framework for creating specialized Large Language Models (LLMs) focused on Earth Intelligence. The core of EVE is EVE-Instruct, a 24 billion parameter model derived from Mi…
-
GoClick model offers lightweight GUI element grounding for on-device AI agents
Researchers have developed GoClick, a novel lightweight vision-language model designed for precise GUI element grounding on resource-constrained devices. Unlike existing large models, GoClick utilizes an encoder-decoder…
-
Rethinking Token Pruning for Historical Screenshots in GUI Visual Agents: Semantic, Spatial, and Temporal Perspectives
Researchers have explored token pruning strategies for GUI visual agents that utilize Multimodal Large Language Models (MLLMs). Their study revealed that background regions in screenshots, often overlooked, can provide …
-
Holo1: New family of GUI automation VLMs powering GUI agent Surfer-H
Researchers have introduced A11y-Compressor, a framework designed to make GUI agent observations more efficient by transforming linearized accessibility trees into structured representations. This method reduces input t…