esp32-iphone-agent
ESP32 iPhone Agent by Henry Yuan (MIT license) is a hardware-integrated automation agent that allows multimodal AI models to operate an iPhone identically to a human user. By plugging an inexpensive (~$7) Seeed Studio XIAO ESP32-S3 board directly into the iPhone's USB-C or Lightning port, the board acts as a native USB HID mouse/keyboard and AirPlay receiver without requiring iOS jailbreaking, enterprise developer profiles, or a host computer.
The agent runs directly within an accompanying iOS app (distributed via TestFlight and Handsy) on the phone itself. It captures the screen, transmits screenshots to frontier multimodal vision models (such as Claude 3.5 Sonnet, GPT-4o, or DeepSeek-VL) to detect UI elements and bounding boxes, and translates actions into low-level USB HID click, swipe, and keystroke signals routed back through the ESP32 hardware.
This architecture bypasses iOS's rigid sandbox constraints, enabling agents to automate native Apple apps, third-party messengers, or system settings. However, users must acquire and flash physical hardware, configure iOS AssistiveTouch and HID pointer settings, and accept latency dictated by multimodal image round-trips and physical touch intervals.
Features
Why choose it
Trade-offs
Maintenance check
- Last commit
- 2026-11-10
- Commits, last 90 days
- 100+
- Releases, last 90 days
- 8
- Latest release
- cli-v0.1.4 (2026-11-08)
- Open issues
- 24
- Recent issues with at least one comment
- 12/19
Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.
Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.
Most discussed open issues
- #115 iOS app: pointer / sensitivity calibration inaccurate — taps and draw strokes miss (fw 0.2.4, Export trace attached)3 comments · 2026-11-07
- #104 相对 HID 点击前校验指针落点,纠偏失败时停止点击3 comments · 2026-11-07
- #103 支持指定时长的 AI 长按与长按后拖拽3 comments · 2026-11-07
- #134 加入计划进度,避免 Jev 重复操作并减少主模型调用2 comments · 2026-11-09
- #106 iOS:模型任务结束后统一执行 reset,恢复自动化遗留的临时状态2 comments · 2026-11-07
Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.
Compatibility
Quick start
Use cases
Alternatives
Comments
No comments yet. Be the first!