AgentIndex icon
AgentIndex
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Submit Tool
ToolsMCP ServersHOTCategoriesTrendingNewBest ListsCompareGuides
Home/
Workflow Automation/
esp32-iphone-agent
esp32-iphone-agent logo

esp32-iphone-agent

Active·★ 5.0k·MIT·Updated 2026-11-16

ESP32 iPhone Agent by Henry Yuan (MIT license) is a hardware-integrated automation agent that allows multimodal AI models to operate an iPhone identically to a human user. By plugging an inexpensive (~$7) Seeed Studio XIAO ESP32-S3 board directly into the iPhone's USB-C or Lightning port, the board acts as a native USB HID mouse/keyboard and AirPlay receiver without requiring iOS jailbreaking, enterprise developer profiles, or a host computer.

The agent runs directly within an accompanying iOS app (distributed via TestFlight and Handsy) on the phone itself. It captures the screen, transmits screenshots to frontier multimodal vision models (such as Claude 3.5 Sonnet, GPT-4o, or DeepSeek-VL) to detect UI elements and bounding boxes, and translates actions into low-level USB HID click, swipe, and keystroke signals routed back through the ESP32 hardware.

This architecture bypasses iOS's rigid sandbox constraints, enabling agents to automate native Apple apps, third-party messengers, or system settings. However, users must acquire and flash physical hardware, configure iOS AssistiveTouch and HID pointer settings, and accept latency dictated by multimodal image round-trips and physical touch intervals.

This write-up is based on the project's README and public repository data. We have not hand-tested the tool. Data as of 2026-11-12.

License
MIT
Latest release
cli-v0.1.4 · 2026-11-08
Last commit
2026-11-11
Contributors
10
Forks
661
Open issues and PRs
24
First published
2026-10-11
npm package
esp32agent 0.1.5 · 1,311 downloads last week

Source: GitHub, npm and PyPI public APIs.

#agent-framework#ai-agent#airplay#automation#computer-use#dsh-plugin#esp32#esp32-s3
$ Install
$ npx -y esp32agent mcp
↗ Visit site★ GitHub
01

Features

01AirPlay screen capture through an ESP32-S3 board.
02USB HID mouse and keyboard actions.
03iPhone app, CLI and MCP integration paths.
02

Why choose it

+True standalone mobile agent: runs directly between iPhone and a $7 ESP32 board without requiring a Mac or PC host.
+No jailbreaking or enterprise certificates required; simulates native USB HID mouse and keyboard signals.
+Cross-application autonomy capable of driving native iOS apps, system settings, and third-party software.
+Open-source MIT codebase with convenient web-based firmware flashing and active TestFlight builds.
03

Trade-offs

!Requires purchasing and flashing dedicated hardware (Seeed Studio XIAO ESP32-S3 board with OTG adapter).
!Requires enabling iOS AssistiveTouch and configuring accessibility cursor settings for mouse emulation.
!Interaction speed is bottlenecked by multimodal vision model inference latency and screen capture intervals.
!External dongle draws power from the iPhone USB-C/Lightning port, increasing battery drain during long workflows.
04

Maintenance check

ActiveChecked: 2026-11-12
Last commit
2026-11-10
Commits, last 90 days
100+
Releases, last 90 days
8
Latest release
cli-v0.1.4 (2026-11-08)
Open issues
24
Recent issues with at least one comment
12/19
Commits per week, last 12 weeks
Commits per week, last 12 weeks: 0, 0, 0, 0, 0, 0, 0, 47, 27, 3, 80, 18

Oldest week on the left; the latest four weeks are orange. Source: GitHub commit activity.

Status rule: archived on GitHub = Archived; last commit within 90 days = Active; 91 to 365 days = Slowing; older = Stale. All figures come from the GitHub API on the date shown.

Most discussed open issues

  • #115 iOS app: pointer / sensitivity calibration inaccurate — taps and draw strokes miss (fw 0.2.4, Export trace attached)3 comments · 2026-11-07
  • #104 相对 HID 点击前校验指针落点,纠偏失败时停止点击3 comments · 2026-11-07
  • #103 支持指定时长的 AI 长按与长按后拖拽3 comments · 2026-11-07
  • #134 加入计划进度,避免 Jev 重复操作并减少主模型调用2 comments · 2026-11-09
  • #106 iOS:模型任务结束后统一执行 reset,恢复自动化遗留的临时状态2 comments · 2026-11-07

Issue titles are quoted from GitHub. Open the link to check whether a problem still applies to the version you use.

05

Compatibility

ios
iOS 18+ app
Verified via docs
esp32
ESP32-S3 hardware
Verified via docs
mcp
CLI MCP integration
Verified via docs
06

Quick start

1
$ npx -y esp32agent mcp
07

Use cases

↳Run a visual agent task on a test iPhone.
↳Drive a phone from a supported coding-agent client.
↳Prototype physical-device automation using an ESP32 board.
08

Alternatives

⚡Top VS:
VSnod★2.3k→VSMika★9.0k→VSAnyModelBot★4.4k→VScapri-ai-desktop★1.7k→
nod logo
nod★ 2.3k
Open source personal agent with its own computer. It chases invoices, preps your calls and cancels trials on real websites, and asks before it acts. Runs on your machine with the model you choose. By n8n.
VSCompare→
Mika logo
Mika★ 9.0k
Mika — 你手机上的私人 AI 助理,替你操作 App、上网、把事办成。An on-device AI agent for Android that operates apps and the web to get things done for you.
VSCompare→
AnyModelBot logo
AnyModelBot★ 4.4k
🤖 Open-source, local-first alternative to Grok Bot, OpenAI dots and Claude Duet. Your own AI bots on any model: GPT, Claude, Gemini, Grok, DeepSeek, Kimi, GLM and 70+ more through one Modelflare key. Reach them in the app, WeChat or Feishu; they remember you, work in the background, run commands and use your computer. Pay per use.
VSCompare→
capri-ai-desktop logo
capri-ai-desktop★ 1.7k
Cyberpunk AI desktop agent & voice companion for Windows. Supports Claude Code, Ollama, DeepSeek, Discord calls & screen automation.
VSCompare→
clawdcursor logo
clawdcursor★ 406
MCP-powered fallback layer that lets AI agents execute tasks through the GUI when APIs, tools, or direct integrations are unavailable. Cross-OS, accessibility-first, local-only.
Touchpoint logo
Touchpoint★ 48
Give your AI agent eyes and hands on any desktop — cross-platform accessibility API with MCP server
VSCompare→
hermes-agent logo
hermes-agent★ 264.2k
The agent that grows with you
VSCompare→
nocobase logo
nocobase★ 25.0k
NocoBase is an open-source AI + no-code platform for building business systems fast. Instead of generating everything from scratch, AI works on top of production-proven infrastructure and a WYSIWYG no-code interface, so you get both speed and reliability.
VSCompare→
See all alternatives →

Comments

Log in to leave a comment

No comments yet. Be the first!

On this page
01Features02Why choose it03Trade-offs04Maintenance check05Compatibility06Quick start07Use cases08Alternatives
onehumanbeing /
esp32-iphone-agent
↗

Hardware-enabled iPhone automation agent using a $7 ESP32-S3 board for physical-level taps, swipes, and typing

5.0k661MIT
Last commit2d ago
StatusActive
CategoryWorkflow Automation
Contributors10
Latest releasecli-v0.1.4
Links
Documentation↗Discussion↗Issues↗Releases↗
Cloud Infrastructure
Ad

Deploy on DigitalOcean

Host your MCP servers, Docker containers, and AI agents with $200 free credit over 60 days.

Claim $200 Free Credit
© 2026 AgentIndex.app|Built by a 10-year iOS Developer.
Best by use caseAboutContactAdvertisePrivacy PolicyTerms of UseQYSGitHubBuy me a coffee ☕

Browse by Category

MCP ServersCode AssistantWorkflow AutomationRAG / Knowledge BaseMulti-AgentBrowser AutomationLLM InfraDev ToolingObservability

Not affiliated with Anthropic, OpenAI or Microsoft.