PowerInfer Dynamic Hot Neuron Thresholding vs Static Inference in LLaMA-70B Code Generation

This report synthesises findings from 11 peer-reviewed papers addressing the following research question: How does PowerInfer's dynamic hot neuron threshold adjustment compare to static inference methods in terms of throughput and memory efficiency when applied to LLaMA-70B on the HumanEval code. This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference. 10 claims were extracted from source literature; 10 were independently verified against retrieved documents. An automated multi-reviewer quality assessment produced a score of 8.7/10. This report is a machine-generated literature synthesis and does not constitute original research.

Research goal: How does PowerInfer's dynamic hot neuron threshold adjustment compare to static inference methods in terms of throughput and memory efficiency when applied to LLaMA-70B on the HumanEval code generation benchmark?

Autonomous literature synthesis. Automated review score: 8.7/10. Full text and citation available at Assignee Research.