Developing an on-device LLM application for Android presented several production challenges, particularly with Google's LiteRT-LM runtime. Issues included the GPU delegate causing the app to be killed due to excessive memory allocation, and failures in loading OpenCL libraries on newer Android versions. Additionally, naive chat templates could serialize JSON into the context, poisoning subsequent turns, and LiteRT-LM bundles were found to be version-fragile, requiring model re-conversion with SDK updates. The runtime also lacked thread safety, necessitating serialization of all native calls to prevent crashes. AI
IMPACT Highlights practical engineering hurdles in deploying on-device LLMs, potentially slowing adoption until runtime stability improves.
RANK_REASON The item details practical implementation challenges and bugs encountered when developing a specific on-device LLM application using a particular runtime, rather than announcing a new model or significant industry-wide development.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →