Semantic Hand: Running Local AI in the Browser for Physics-Based Hand Control
A new AI sandbox called Semantic Hand allows users to control a virtual hand in a browser environment using natural language prompts. The system leverages local AI models like Nemotron 3 Nano 4B or Qwen 3.5-4B, running through Transformers.js and WebGPU, to interpret user commands and translate them into physical actions for the virtual hand. Challenges encountered included refining system prompts to prevent hallucinations and ensuring the AI properly considered object shapes and physics, rather than just point-based interactions. AI
IMPACT Enables interactive AI demonstrations and research directly within web browsers, lowering barriers for experimentation with AI-driven physical simulations.