Apple’s on-device AI development: Expected, or surprise?

TLDR: Apple is developing its own on-device AI, which should come as no surprise given the company’s focus on machine learning algorithms for subtle improvements to the user experience. Apple has been quietly buying AI companies and developing its own large language models to ensure it can deliver generative AI capabilities. The company’s homebrew silicon, which includes neural processing units (NPUs) called “Neural Engines,” already handle smaller machine learning workloads on iPhones. The NPUs in Apple’s latest iPhone models are particularly powerful, capable of pushing 35 TOPS of performance. While computational power isn’t an issue for running larger AI models on Apple’s devices, memory constraints may pose a challenge. AI models typically require about a gigabyte of memory for every billion parameters, and Apple may need to increase the memory on its phones or develop smaller, more targeted models. However, there is evidence to suggest that Apple has found a way to address memory limitations by running large language models on-device using flash memory. When Apple introduces AI functionality on its devices, it is expected to take a conservative approach, likely starting with improvements to its virtual assistant Siri.