Breaking the Memory Wall: Chiplet Architectures for the AI Era
Researchers are developing heterogeneous memory chiplets and long-span ECC controllers to overcome the 'memory wall' in LLM inference. These semiconductor innovations aim to reduce overhead and accelerate AI processing at the hardware level.
The semiconductor industry is currently locked in a battle against the "memory wall"—the bottleneck where data movement between processors and memory consumes more time and energy than the computation itself. To solve this, researchers at the National University of Singapore have introduced "CHIPSMORE," a compute-in-interconnect and -memory chiplet architecture designed specifically for large language model (LLM) inference. By integrating processing capabilities directly into the memory interconnect, the system can handle multi-request AI tasks with significantly lower latency.
Simultaneously, a collaboration between IBM and Rensselaer Polytechnic Institute has resulted in "REACH," a long-span Error Correction Code (ECC) controller managed specifically for High Bandwidth Memory (HBM). In high-performance AI inference, the overhead of error correction can become a significant drag on performance. REACH reduces this overhead by managing ECC at a broader span, ensuring data integrity without the traditional speed penalties. This is critical as AI models grow in complexity and require ever-higher reliability at the hardware level.
Beyond memory, the move toward next-generation lithography is gaining steam. TSMC and National Yang Ming Chiao Tung University have published findings on molybdenum quasi phase-only masks to improve Extreme Ultraviolet (EUV) imaging. These topological masks enable higher contrast in chip manufacturing, allowing for the further miniaturization of transistors. These disparate yet connected innovations in chiplet design, error correction, and lithography are the foundation upon which the next generation of Physical AI and autonomous systems will be built.
Source: Semiconductor Engineering
