目录：

UI Agents 知识星球UI Agents 综述资料【Update 2024.12.19】UI Agents 论文列表【Update 2024.12.23】Policy ModelsTraining-based ModelsTraining-free ModelsEnhanced KnowledgesData SynthesisDatasets / BenchmarksTools / EnvironmentsUI Agents 其他汇总信息

UI Agents 知识星球

加入知识星球，每周获取会员专享视频👇

 

扫码加微信小助手为好友，备注「agent」，小助手会定期邀请入群👇

UI Agents 综述资料【Update 2024.12.19】

• UI Agents（智能体）技术综述, Bilibili, Youtube, Breezedeus, 2024.11

• [2412.13501] GUI Agents: A Survey

• [2412.10047] Large Action Models: From Inception to Implementation, Microsoft

• [2411.18279] Large Language Model-Brained GUI Agents: A Survey, Microsoft

• [2411.10323] The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use

• [2411.04890] GUI Agents with Foundation Models: A Comprehensive Survey, Huawei Noah’s Ark Lab

• [2411.02006] Foundations and Recent Trends in Multimodal Mobile Agents: A Survey

UI Agents 论文列表【Update 2024.12.23】

Policy Models

Training-based Models

• [2412.09362] Falcon-UI: Understanding GUI Before Following User Instructions

• [2412.04454] Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

• [2412.01268] Ponder & Press: Advancing Visual GUI Agent towards General Computer Control

• [2411.17465] ShowUI: One Vision-Language-Action Model for GUI Visual Agent, Github

• [2402.15506] AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning

• [2411.13451] AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

• [2411.00820] AutoGLM: Autonomous Foundation Agents for GUIs, Zhipu

• [2410.19461] EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

• [2410.22916] Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration

• [2406.19263] Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

• [2412.10742] WEPO: Web Element Preference Optimization for LLM-based Web Navigation

• [2412.06313] Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information

• [2411.03817] From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning

• [2411.02337] WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning, Zhipu

• [2410.24218] Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use

• [2406.11896] DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Training-free Models

• [2412.10840] Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

• [2412.07472] SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

• [2411.15004] ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

• [2411.06559] Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

• [2408.06458] Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

• [2411.13591] Improved GUI Grounding via Iterative Narrowing

• [2410.19609] OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

• [2408.00203] OmniParser for Pure Vision Based GUI Agent

• [2407.03913] MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

• [2405.15341] V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM

• [2312.11190] VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning

Enhanced Knowledges

• WMA[2410.13232] Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

• [2410.23555] From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents

• [2410.22552] Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents

• [2405.16247] AutoManual: Generating Instruction Manuals by LLM Agents via Interactive Environmental Learning

Data Synthesis

• [2412.10342] Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

• [2412.09605] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Datasets / Benchmarks

• [2410.17520] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

• [2412.04531] MageBench: Bridging Large Multimodal Models to Agents

• [2410.06703] ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

• [2410.24024] AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents, Zhipu

• [2410.15164] SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

• [2410.17520] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

• AndroidControl[2406.03679] On the Effects of Data Scale on UI Control Agents

• [2406.14250] E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

• [2405.14573] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Tools / Environments

• [2412.05789] InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction

• [2412.05467] The BrowserGym Ecosystem for Web Agent Research

• [2403.17918] AgentStudio: A Toolkit for Building General Virtual Agents

UI Agents 其他汇总信息

• https://github.com/OSU-NLP-Group/GUI-Agents-Paper-List

• https://github.com/showlab/Awesome-GUI-Agent

• https://github.com/opendilab/awesome-ui-agents