During the highly anticipated WWDC25, Apple unveiled exciting advancements in its groundbreaking on-device and cloud-based foundation models. Following this announcement, the tech giant has released a comprehensive tech report that elaborates on how these models were meticulously trained, optimized, and evaluated. The document, titled “Apple Intelligence Foundation Language Models – Tech Report 2025”, delves into various essential aspects of the new models, including their architecture, data sources, pre-training methodologies, post-training adjustments, and performance benchmarks. For those interested in the intricate details of AI development, this technical report is an invaluable resource.
Among the noteworthy insights in the report, one particular focus is on the structure of Apple’s local model. Previously known to contain approximately 3 billion parameters, Apple has revealed that this model is divided into two distinct blocks. Specifically, Block 1 incorporates 62.5% of the total transformer layers, while Block 2 comprises the remaining 37.5% but omits the key and value projections. This strategic division results in a 37.5% reduction in memory requirements for caching, significantly expediting the time taken to output the first token, a crucial aspect of performance. Apple asserts that this innovative architecture preserves the overall performance and output quality of the model.
Interestingly, Apple previously explored the concept of swapping portions of a large language model (LLM) between RAM and flash storage to optimize local model performance. Although the current implementation diverges from that approach, it highlights Apple’s ongoing commitment to enhancing local performance, particularly on devices with limited memory capacity.
Turning to Apple’s cloud-based model, the report introduces a custom architecture developed specifically for its Private Cloud Compute platform, known as Parallel-Track Mixture-of-Experts (PT-MoE). This novel architecture employs a unique strategy where the massive AI model is partitioned into smaller subnetworks, or experts, activated based on the relevance to the task at hand. For instance, if a user prompt pertains to culinary topics, only the cooking-related experts are engaged, keeping other areas dormant. This architecture not only maintains a substantial overall model size but also enhances response speed and accuracy by preventing all components from running simultaneously.
Apple has redefined the traditional transformer architecture by introducing the Parallel Track Transformer, which scales up through Mixture of Experts (MoE) layers. In essence, instead of processing tokens in a single sequence, Apple’s design allows for multiple parallel tracks that independently process tokens and synchronize at designated intervals. Within these tracks, every alternate transformer layer is replaced with an MoE layer, activating only a few pertinent experts for each token, thus alleviating processing bottlenecks and enabling a more efficient system.
One of the major criticisms of Apple Intelligence has been its limited support for languages beyond English. In response, Apple has made significant strides in increasing multilingual representation, raising the amount of multilingual data utilized in training from 8% to an impressive 30%. This includes both organic and synthetic content, and Apple has also augmented its tokenizer by 50%, increasing its vocabulary to 150,000 tokens from the previous 100,000. These enhancements have reportedly resulted in substantial performance improvements across non-English benchmarks, particularly after fine-tuning through reinforcement learning.
The report clarifies that evaluations were conducted using prompts generated by native speakers, rather than relying on translations, allowing for a more accurate assessment of the model's performance in local contexts. This translates into more reliable functionality for features like Writing Tools in supported languages, enhancing user experience significantly.
As with its initial models, Apple primarily sourced training data through web crawling. Notably, Apple's Applebot crawler adheres to robots.txt exclusions, ensuring that sites wishing to protect their content can do so. The report outlines the various data sourcing methods that contributed to the development of the new models:
Publicly Available Web Data: The majority of training data was derived from Applebot’s web crawls, which underwent rigorous filtering to eliminate low-quality or irrelevant content. Licensed Data: Although specifics are scarce, Apple confirms that some training data was licensed from reputable publishers, including potential partnerships with Condé Nast and NBC News. Synthetic Data: Apple generated synthetic data via smaller models, especially for tasks related to math, code, and vision-language, playing a critical role in fine-tuning and multilingual support. Visual Data: To enhance image understanding, Apple collected over 10 billion image-caption pairs, combining various approaches to enrich the dataset further.Despite ongoing challenges and perceptions of lagging behind competitors in the AI space, Apple's recent tech report reveals a wealth of under-the-hood improvements and innovative strategies aimed at enhancing its AI offerings. This report not only highlights the company’s technical progress but also showcases its commitment to privacy-conscious practices that few firms are attempting. As Apple continues to evolve its AI capabilities, the insights gleaned from this report provide a compelling look at the future of its AI technologies.
