Optimizing Local LLM Inference on Apple Silicon: A Deep Dive into Performance Gains

Recent research has unveiled significant insights into optimizing large language model (LLM) inference specifically on Apple Silicon—popular for its efficiency in computational tasks. The study, conducted by a team from MacPaw AI&Research, meticulously evaluates various optimization techniques to determine their effectiveness on Apple's unique architecture.

Apple Silicon: The Rising Star of Local LLM Inference

As local LLM inference gains traction, Apple Silicon emerges as a top platform choice due to its innovative architecture and capabilities. Despite its rise, existing optimization techniques have predominantly targeted server-class NVIDIA GPUs, leaving a notable gap regarding applicability on Apple’s hardware. This prompted researchers to explore which techniques enhance performance when operating on Apple Silicon.

Exploring Optimization Techniques

The research focuses on four primary optimization families:

• KV-cache Pruning: This technique aims to enhance throughput by reducing the data processed during inference.
• Speculative Decoding: This method attempts to accelerate the LLM inference process by efficiently predicting sequences.
• Metal Kernel Porting: Many established CUDA techniques were adapted for Metal—a graphics and compute API specific to Apple hardware.
• Post-Training Quantization: A method that reduces the model size and computational requirements, allowing smoother inference performance.

Key Findings and Performance Insights

Each optimization was rigorously evaluated across various Apple devices including M2 Pro, M3 Ultra, and M4 Pro, with comparisons made against NVIDIA RTX 4090 controls where possible. The study highlighted:

• KV-cache Pruning: This method yielded impressive results on Apple Silicon, achieving up to 1.75x end-to-end throughput. Yet, it showed diminished effectiveness on NVIDIA GPUs.
• Speculative Decoding: Contrary to expectations of accelerating performance, all tested methods resulted in reduced throughput on Apple, revealing the complexities of adapting speculative decoding to new architectures.
• Metal Ports: Although certain components from CUDA porting to Metal showed promise, the performance gains were inconsistent, attributed to incompatibilities with Apple’s handling of asynchronous operations.
• Quantization Techniques: Performance benefits from quantization varied significantly across devices, illustrating the importance of evaluating optimizations within specific hardware contexts.

Importance of Contextual Optimization

The research underscores a vital takeaway: the effectiveness of an optimization technique is not a uniform attribute, but rather a product of the combination of technique, runtime, hardware, and the specific workload at hand. This complex interplay is crucial for developers seeking to optimize LLMs for various platforms.

Looking Forward

The investigation lays a foundation for future explorations into local LLM inference on Apple Silicon, spotlighting the need for continuous adaptation of existing methodologies to fit emerging technologies. It highlights the promising potential of Apple Silicon while also inviting further discussions around optimizing performance across different hardware architectures.

In summary, understanding how various optimization techniques perform on Apple Hardware is key to harnessing the full capabilities of LLMs, making this research a significant contribution to the AI and machine learning communities.

Authors: {Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn, Ostap Khomenko, Pavlo Kryven, Maksym Kmet, Maksym Shamrai, Mariya Hirna}